Storage interface for shared strings

[xakep666]

Thanks for nice library. Currently I'm using it to parse large spreadsheets (500k rows and more) that contains data for mailing. Currently shared strings are fully loaded to memory during XlsxFile initialization (io.ReadAll + xml.Unmarshal + getPopulatedValues). When nearly all strings in spreadsheet are unique this technique leads to huge memory consumption.
My idea is:

• add options functional to constructors in compatible way i.e. NewReader([]byte, ...Option)
• first option will be optional shared strings storage interface like

type SharedStringStorage interface {
  SetSharedString(idx int, string) error
  SharedString(idx int) (string, error)
}

By default parser can use simple implementation based on []string or map[int]string

• read shared strings in streaming mode and store them using provided interface

If there is no concerns about ideas above I'll try to make pull request.

[dglsparsons]

Hi, thanks for the message - it's great to see people using the library and contributing back :).

I can totally understand how large SharedStrings files could cause an issue here. Do we know if it's the XML parsing that causes huge memory consumption, or if is a result of storing the sharedStrings in memory?

If it's the former, could we stream the shared strings file when parsing (as we do with the main XML content) - I'm not sure I see the need for the interface changing here.

If it's the latter, I'm a bit confused how your suggestion would resolve the issue?

[xakep666]

Hm, you're may be right. alloc_space measurements

shows that during shared string parsing memory allocated for raw file bytes, parsed sharedString.Values and []string. So really, as good first step streaming parsing of shared string entries just to []string may be implemented.
New storage interface may be useful for files that contains millions of strings (i.e. implemented by something like lru cache with swapping of unused items to disk).

[dglsparsons]

That's a great measurement - clearly highlights the issue we have :)

An LRU cache will likely add a tonne of complexity, especially around re-streaming and parsing the XML content, but retaining cached values, etc. It looks like it's unlikely to be necessary either - at least in the short term.

I think the simplest thing, for now, is simply to update the XML parser into one that steams when parsing shared strings. From there, we can always see what the memory issues are like and if any further work would be needed.

Great work and nice discovery!

[xakep666]

@dglsparsons I've made pull request #31 that implements streaming decode of shared strings

[xakep666]

@dglsparsons @AndyThurgood thanks for merge! Is it possible to tag new release for usage in go.mod?