Handle uploads > 4GiB #22
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Currently Wasm modules are limited to a 32bit address space (until at
least the memory64 work is completed). All the counters etc in the
request structure were u32's. Which matched with the 32bit memory
limitation.
However there is really no need to not allow >4GiB uploads that can be
saved off to disk or some such.
To do this we need to increase the ->content_len & ->total_content_sent
members to u64's and also adjust the return types of
(luw,uwr}_get_http_content_len() and
{luw,uwr}_get_http_total_content_sent() similarly.
However because we need the request structure to have the exact same
layout on 32bit (for Wasm modules) as it does on 64bit we need to re-jig
the order of some of these members and add a four-byte padding member.
Thus the request structure now looks like on 32bit (as shown by
pahole(1))
struct luw_req {
u32 method_off; /* 0 4 */
u32 method_len; /* 4 4 */
u32 version_off; /* 8 4 */
u32 version_len; /* 12 4 */
u32 path_off; /* 16 4 */
u32 path_len; /* 20 4 */
u32 query_off; /* 24 4 */
u32 query_len; /* 28 4 */
u32 remote_off; /* 32 4 */
u32 remote_len; /* 36 4 */
u32 local_addr_off; /* 40 4 */
u32 local_addr_len; /* 44 4 */
u32 local_port_off; /* 48 4 */
u32 local_port_len; /* 52 4 */
u32 server_name_off; /* 56 4 */
u32 server_name_len; /* 60 4 */
/* --- cacheline 1 boundary (64 bytes) --- */
u64 content_len; /* 64 8 */
u64 total_content_sent; /* 72 8 */
u32 content_sent; /* 80 4 */
u32 content_off; /* 84 4 */
u32 request_size; /* 88 4 */
u32 nr_fields; /* 92 4 */
u32 tls; /* 96 4 */
char __pad[4]; /* 100 4 */
struct luw_hdr_field fields[]; /* 104 0 */
/* size: 104, cachelines: 2, members: 25 */
/* last cacheline: 40 bytes */
};
and the same structure (taken from Unit) compiled as 64bit
struct nxt_wasm_request_s {
uint32_t method_off; /* 0 4 */
uint32_t method_len; /* 4 4 */
uint32_t version_off; /* 8 4 */
uint32_t version_len; /* 12 4 */
uint32_t path_off; /* 16 4 */
uint32_t path_len; /* 20 4 */
uint32_t query_off; /* 24 4 */
uint32_t query_len; /* 28 4 */
uint32_t remote_off; /* 32 4 */
uint32_t remote_len; /* 36 4 */
uint32_t local_addr_off; /* 40 4 */
uint32_t local_addr_len; /* 44 4 */
uint32_t local_port_off; /* 48 4 */
uint32_t local_port_len; /* 52 4 */
uint32_t server_name_off; /* 56 4 */
uint32_t server_name_len; /* 60 4 */
/* --- cacheline 1 boundary (64 bytes) --- */
uint64_t content_len; /* 64 8 */
uint64_t total_content_sent; /* 72 8 */
uint32_t content_sent; /* 80 4 */
uint32_t content_off; /* 84 4 */
uint32_t request_size; /* 88 4 */
uint32_t nfields; /* 92 4 */
uint32_t tls; /* 96 4 */
char __pad[4]; /* 100 4 */
nxt_wasm_http_field_t fields[]; /* 104 0 */
/* size: 104, cachelines: 2, members: 25 */
/* last cacheline: 40 bytes */
};
We can see the structures have the same layout, same size and no
padding.
We need the __pad member as otherwise I saw gcc and clang on Alpine
Linux automatically add the 'packed' attribute to the structure which
made the two structures not match.
Link: <https://github.com/WebAssembly/memory64>
Signed-off-by: Andrew Clayton <[email protected]>
The previous commit changed uwr_get_http_content_len() to return a u64 to allow for uploads larger than 4GiB, which now means this generates compiler errors about type mismatches, expected usize got u64. Cast the return value of uwr_get_http_content_len() to usize to match that of TOTAL_RESPONSE_SENT. (Making TOTAL_RESPONSE_SENT a u64 creates a larger trail of problems). Signed-off-by: Andrew Clayton <[email protected]>
This is analogous to luw_req_buf_append() but rather than appending
request data to the buffer it simply overwrites what's currently there.
This is needed to take advantage of the new ability to receive >4GiB
requests/payloads.
On a new request you would call luw_init_ctx(), luw_set_req_buf() &
open(2).
On subsequent calls to the request_handler (for this same HTTP
request/upload) you would call this new function and then write out the
data to a file.
E.g
/* In the request_handler */
if (total_bytes_wrote == 0) {
luw_init_ctx(&ctx, addr, 0);
luw_set_req_buf(&ctx, &request_buf, LUW_SRB_NONE);
fd = open("/var/tmp/large-file.dat", O_CREAT|O_TRUNC|O_WRONLY,
0666);
} else {
luw_req_buf_copy(&ctx, addr);
}
buf = luw_get_http_content(&ctx);
bytes_wrote = write(fd, buf, luw_get_http_content_sent(&ctx));
total_bytes_wrote += bytes_wrote;
if (total_bytes_wrote == luw_get_http_content_len(&ctx)) {
close(fd);
total_bytes_wrote = 0;
luw_http_response_end();
}
Signed-off-by: Andrew Clayton <[email protected]>
This is inspired by the likes of splice(2) and sendfile(2) in that it
takes data from one place and puts it in another.
This function write(2)'s the request data straight from the shared
memory to a given file (referenced by its file descriptor).
This is an alternative to using luw_req_buf_copy() and avoids an extra
copying of the request data.
E.g
/* In the request_handler */
if (total_bytes_wrote == 0) {
luw_init_ctx(&ctx, addr, 0);
luw_set_req_buf(&ctx, &request_buf, LUW_SRB_NONE);
fd = open("/var/tmp/large-file.dat", O_CREAT|O_TRUNC|O_WRONLY,
0666);
}
total_bytes_wrote += luw_mem_splice_file(addr, fd);
if (total_bytes_wrote == luw_get_http_content_len(&ctx)) {
close(fd);
total_bytes_wrote = 0;
luw_http_response_end();
}
NOTE:
We include a typedef definition for ssize_t in unit-wasm.h, to avoid
having a dependency on the wasi-sysroot when generating the rust
bindings.
ssize_t is defined in sys/types.h which is provided by libc and not the
compiler.
Signed-off-by: Andrew Clayton <[email protected]>
libunit-wasm added two new functions, luw_req_buf_copy() & luw_mem_splice_file(). See the previous two commits... This second function takes a file-descriptor as one of its arguments, in rusty we make this a Rust File object, then pass the underlying fd into libunit-wasm. Signed-off-by: Andrew Clayton <[email protected]>
The types of some of the member of the luw_req structure increased to 64bits to allow for uploads larger than 4GiB. Two new functions were added luw_req_buf_copy() Like luw_req_buf_append() but just copies the data over what's already there. luw_mem_splice_file() This write(2)'s the request data directly from the shared memory to a given file. Signed-off-by: Andrew Clayton <[email protected]>
uwr_get_http_content_len() & uwr_get_http_total_content_sent() now return a u64. Two new functions were added uwr_req_buf_copy() Like uwr_req_buf_append() but just copies the data over what's already there. uwr_mem_splice_file() This write(2)'s the request data directly from the shared memory to a given file. Signed-off-by: Andrew Clayton <[email protected]>
The programs demonstrate handling requests with payloads larger than 4GiB which means they need to be written out to disk and so also demonstrates the use of the file-system access mechanism. Signed-off-by: Andrew Clayton <[email protected]>
|
Rebase with master |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This patch set adds the ability to handle requests larger than 4GiB (which is larger than the 32bit address space we're limited to).
This is accomplished by increasing a couple of the request structure members to 64bits and the addition of a couple of functions
luw_req_buf_copy()
Like luw_req_buf_append() but rather than appending request data to the buffer it simply overwrites what's currently there.
luw_mem_splice_file()
An alternative to luw_req_buf_copy() and avoids an extra copying of the request data. This is inspired by the likes of splice(2) and sendfile(2) in that it takes data from one place and puts it in another.