Caching support

[talentlessguy]

Hello, I have a question regarding caching. Is it supported, e.g. saving stuff to files not to make new requests?

Atm I'm using node-fetch-cache but would like to switch to httpie

[lukeed]

Hey,

This isn't supported and likely wouldn't be since there'd be no cross-platform solution (httpie@next exists for Workers, browsers, and Node.js). Definitely sounds like another module's target, but here's a way you could force/fake it with httpie:

const fs = require('fs');
const { join } = require('path');
const { send } = require('httpie');

const CACHE = join(__dirname, '...');

function toFilename(method, path) {
  return `${method}__${encodeURIComponent(path)}.txt`;
}

async function isFileFresh(file) {
  let now = Date.now();
  let stats = await fs.stat(file);
  return now < +stats.mtime; // custom "mtime" value
}

async function write(file, res) {
  const { statusCode, headers, body='' } = res;
  const output = { statusCode, headers, body };
  
   // spec-compliance: status range, Cache-Control values, etc
   if (!can_cache(statusCode, headers)) return output;

   await fs.promises.writeFile(file, JSON.stringify(output));

   const secs = get_maxage(headers['cache-control']);
   const now = Date.now(), expires = now + secs * 1e3;
   // use `mtime` for cache-expiration date
   // ~> only auto-set via mknod, write, utimes methods (1)
   await fs.promises.utimes(file, now, expires);

  return output;
}

async function fetch(method, path, headers={}, body='')  {
  // checking file system 
  let file = join(CACHE, toFilename(method, path));

  // spec-compliant checks; eg "no-cache" header?
  if (can_reuse_cache(headers) && fs.existsSync(file) && await isFileFresh(file)) {
    return read_and_parse(file); // read file, sending { status, headers, body }
  }
  
  return send(method, path, { headers, body }).then(res => {
    return write(file, res);
  });
}

(1): https://nodejs.org/api/fs.html#fs_stat_time_values

^ This is completely untested, haha. All functions with underscores are left unimplemented intentionally – their needs are well-defined via MDN/specs. The read_and_parse method would need to cast a JSON blob back into JSON, for example. And some response types might break if you JSON.stringify them – so might want to find a different fs approach than shoving it all into a single file, haha. Not 100% sure!

If I ever run into this need and have to make something, I'll try to remember to circle back here. But for the most part, I just rely on network cache and the browsers' local caches.

Hope that helps~

[talentlessguy]

@lukeed thanks for a detailed response!

btw as a thought, there could be a separate file for node with caching, that could be used like this:

import httpie from 'httpie/caching'

Using exports field

(just a thought)

but I agree it better be a separate module

[lukeed]

I have separate entries already, and they're all focused on being API-compatible with one another. It wouldn't make sense to add something that one supported one platform.

Plus, I forgot to mention earlier, but httpie wouldn't be super ideal for this (in current form) cuz everything is stashed in memory before returning and/or writing to file. Something like a raw function (#25) would have to be added first to support streams and I'm still not sure where I stand on that for this project.