Switch dev Command to Use llamafile
#559
Conversation
| @@ -4,7 +4,7 @@ | |||
| // you may not use this file except in compliance with the License. | |||
| // You may obtain a copy of the License at | |||
| // | |||
| // http://www.apache.org/licenses/LICENSE-2.0 | |||
There was a problem hiding this comment.
I think some formatter replaced these spaces with a tab
There was a problem hiding this comment.
go fmt does it somehow inconsistently
.goreleaser.linux.yaml
Outdated
| @@ -31,6 +42,20 @@ archives: | |||
| files: | |||
| - LICENSE | |||
| - README.md | |||
| - id: kit-archive-offile | |||
There was a problem hiding this comment.
| - id: kit-archive-offile | |
| - id: kit-archive-offline |
| OWNER=$1 | ||
| REPO=$2 | ||
| RELEASE_TAG=$3 | ||
| ASSET_NAME=$4 | ||
| OUTPUT_DIR=$5 | ||
|
|
||
| echo "Downloading asset '${ASSET_NAME}' from release '${RELEASE_TAG}' of repository '${OWNER}/${REPO}'" | ||
|
|
||
| DOWNLOAD_URL="https://github.com/${OWNER}/${REPO}/releases/download/${RELEASE_TAG}/llamafile-${RELEASE_TAG}" | ||
|
|
||
| mkdir -p ${OUTPUT_DIR} | ||
|
|
||
| curl -L -o ${OUTPUT_DIR}/llamafile ${DOWNLOAD_URL} | ||
|
|
||
| echo "Asset '${ASSET_NAME}' downloaded successfully to: ${OUTPUT_DIR}/${ASSET_NAME}" | ||
|
|
||
| echo "${RELEASE_TAG}" > ${OUTPUT_DIR}/llamafile.version | ||
|
|
||
| COMPRESSED_FILE="llamafile.tar.gz" | ||
| compress ${OUTPUT_DIR} ../${COMPRESSED_FILE} | ||
|
|
||
| echo "Compressed asset saved to: ${COMPRESSED_FILE}" | ||
| } |
There was a problem hiding this comment.
The usage of $ASSET_NAME is not clear here -- it looks like it's actually hard-coded to llamafile in commands. Maybe the argument can just be dropped?
There was a problem hiding this comment.
The idea was we could use it to download other binaries or checksums. We did not need that at the moment but kept it just in case.
| cmd = exec.Command("sh", "-c", | ||
| fmt.Sprintf("./llamafile --server --model %s --host %s --port %d --path %s --nobrowser --unsecure", | ||
| modelPath, harness.Host, harness.Port, uiHome), | ||
| ) |
There was a problem hiding this comment.
Does this need to be wrapped in a sh -c?
There was a problem hiding this comment.
Unfortunately, because of the way llamafile works.
| _, err = io.Copy(out, resp.Body) | ||
| if err != nil { | ||
| return fmt.Errorf("failed to write to file %s: %w", filePath, err) | ||
| } |
There was a problem hiding this comment.
If the response includes a size for the body, you could probably get a decent-looking progress bar by wrapping resp.Body via WrapReadCloser (though the info line would probably be wrong
Uses llamafile for harness functionality. Introduces downloading the harness in addition to embedding to allow for smaller binaries. Removes llama.cpp submodule Adds script for uploading llamafile to downloads.
Updates release scripts to include both embedded (offline) and download version.
Adds checksum download and verify. Also minor improvements to error handling
Removes the wins/linux restriction. Fixes the archive package for wins offline.
Fix the windows build archive and the exec command.
Minor updates to address review comments
This pull request updates the
devcommand to utilizellamafile. It introduces two methods for packaging the harness:Download Mode (Default): Downloads harness components dynamically first time the dev command is invoked.
Offline Mode: Prepackages harness components for offline use, allowing development without internet access.
Introduces a new build tag
embed_harnessto compile binaries specifically for offline mode. This mode embeds the llamafile and UI assets directly into the binaryModifies the existing build process to generate both default and offline mode binaries. Produces separate archives for each mode.
go generate command downloads the llamafile for the specified version. Prepares the downloaded llamafile for both embedding (offline mode) and uploading (download mode).
Introduces a new script to streamline the uploading of llamafile binaries to the designated download location.
What is missing
Related Issues