Noticed during #1007, looks like the pointer stored by `BufferInner` is never used. We should check we aren't leaking a buffer or anything like this.