What follows documents futility. Effort expended for nothing.
It began when I (greedy for a more diverse vocabulary) expanded the corpus somewhat erratically.
Added my tumblr blog jhavelikes.tumblr.com (for contemporary tech words), then Kraznahorkai’s War And War, Maurice Blanchot’s The Space of Literature, some Kenneth Patchen, and a set of contemporary poetry syllabus packages for added breadth.
Corpus swelled to 65.3mb
Tried it on Wavenet. After 24 hours got gibberish, no idea why, convergence not occuring, spastic spikes of loss disrupting system.
So shifted back to Pytorch.
And using 1500 embedded layers began crashing :
RuntimeError: cuda runtime error (2) : out of memory
Eventually reduced layers to 500. It runs.
jhave@jhave-Ubuntu:~/Documents/Github/pytorch-poetry-generation/word_language_model$ python main_June2017.py --cuda --data=data/dec_rerites --emsize=500 --nhid=500 --dropout=0.65 --epochs=80 --tied INITIALIZING Directory: models/2017-12-03T18-34-4
Even with cuda enabled on a TitanX GPU, training is achingly slow: 17,811 batches at about 500ms per batch + validation time means a single epoch takes more than 3 hours to complete. It needs maybe 40? 60? 100? epochs to arrive anywhere interesting?
Wait a couple days. Sigh. Still not finished.
Wait another 2 days. Sigh. Still not finished.
Stop it anyway.
Run it.
First thing I notice is, it’s very slow. Big corpus slows down generation times as well as training time. Second thing: it’s not appreciably better. Third, the contemporary lexicon (scientific, net-based) that I had hoped to induce into the poetry, massaging it forward from the 15th century toward the 21st, is imperceptible.
Result: throw it all away. Start again. Reduce corpus size to 2.8mb and run with 2000 hidden layers. Wait some more… Am waiting now ….