How to train with a specific set of features/games

[YamilVidal]

Hi all!!
As I experiment with different hyperparameters, I would like to be able to train different networks with exactly the same feature-labels, so that any different in performance would be due to the hyperparameters and not the data itself.

Similarly, I have my own collection of historical go games, and would like to be able to train networks with those games instead of the KGS games.

As I'm relatively new to Python, the task of modifying the existing code (parallel_processor.py, generator.py, etc) is quite daunting. So before I give it a try myself I rather ask. Has anyone implemented something in this line, and would be willing to share it?
Thanks a lot

[macfergus]

Hi Yamil, I have some examples in another repository that might help you.

https://github.com/macfergus/badukai

Here are some key files:
https://github.com/macfergus/badukai/blob/master/build_index.py 👈 builds an index of your SGF collection
https://github.com/macfergus/badukai/blob/master/train_from_sgf.py 👈 handles training over a collection that may be too large to fit in memory
https://github.com/macfergus/badukai/blob/master/badukai/corpora/index.py 👈 module that handles iterating over a large game collection

The differences from the processor in the book are:

• The book processor downloads the KGS SGFs for you; the badukai code just reads them from your disk. So if you already have a game collection, you can just point it there
• The book processor does random downsampling to fit things in memory, the badukai code uses the whole corpus
• The badukai code handles breaking up a large corpus into deterministic chunks that fit in memory
• The badukai code lets you stop training and resume where you left off

I wrote the badukai code after finishing the book, so it uses similar conventions to the book, but not identical. I hope this makes it easier to follow rather than harder 😅. Let me know if this helps!

[YamilVidal]

This is great! Thank you
If I get stuck I'll let you know =)