PyTorch Poetry Generation [Pre-WordHack : Epoch 16 Video]

Another day of testing before going to NYC to perform neural-net poems at WordHack [NYC (Thursday 2/16/2017 @ Babycastles . 7-10pm) w. Sarah Rothberg, John Cayley and Theadora Walsh]


 In the cold weather going out of the snow, 
 She down the lawn. 
 The air moves and grows, while she walks smooth, 
 When a swan is born, 
 And it's almost happening 
 Who knows what to say 
 The change has brought 
 Throwing the first blood in its face.

It’s clear:

Never will this mode of randomized pattern-reasoning replicate the nuanced human heart. More robust ensemble methods that simulate embodied experience, temporal reflexes, and nested community idioms will be required.

Deep learning is still shallow. The cloud does not understand honey, home or heart. Yet in the short-term, this is the future of writing: a computational assistant for an engaged imagination intent on exploring the topological feature-space of potential phrases.


Modulated the parameters: raised both embedding size and hidden layers to 512. And did a bit more data mining and parsing to increase the corpus size by 1/3 to 20mb of .txt.

Mode: LSTM
Embedding size: 512
Hidden Layers: 512
Batch size: 20

Expanded Corpus to over 600,000 lines

639,813 lines of poetry from 5 websites.

Poetry Foundation
Capa - Contemporary American Poetry Archive
Evergreen Review
Shampoo Poetry

After only 1 epoch

… of training (basically the system has read the corpus once and created a model based upon that reading, a process that takes about 12 minutes on TitanX), the system spits out semi-plausible dada-juice:

Epoch: 1
Loss: 6.38


 The bones of my Chasms 

 Round the eye with day what hair 
 Is in the waves that wrote us 
 To a black blade, of vowel fed.

After 2 Epochs

… the juice gets sporadically slightly thicker:

Epoch: 2
Loss: 6.27

 My feet grow lattices where their horses 
 rise to their lips. The girls go Vanish, up 
 & left twenty murmur'd go out of my imaginary

By Epoch 3

… things continue to unravel resiliently, even as the validation loss increases slightly, the testing loss is descending, which may of course signal the dreaded overfitting, — but this being poetry and not finance, i do not care as long as the system produces lines which never existed before.

In this instance, overfitting may mean that the model is becoming capable of speaking in any style or hybrid mashup glomp :

Epoch: 3
Loss: 6.36

 A yellow brown darkens 
 Must come, no shape 
 Than low undone, 
 Beside the water enraptured-- 
 hungry; of youngish found 
 The couch'd fox-head 
 within its harp, 
 Came from the earth uncivil; 
 So the moss were mixed 
 And bent off

At Epoch 4

…validation loss and perplexity are even worse! But the test loss and perplexity (loss  4.83 | ppl   125.05 ) are way better. Let’s dive into that sculpted overfit snug wash of poetry, it seems as if the model has become besotted and fallen in love with love:

Epoch: 4
Loss: 6.42

 I am the torch a letter is lov'd out, 
 with a white saint scrolls from one hand in the driveway. 
 My blazing soul is hungry the tongue 
 can tell the brusque King of the Lord Oldsmobile 


 I saw as proclaimed: with his drape. 
 I put a straight mine for my sunflower 
 She moved; the knee and slow his foot 
 Shows to each stroke of things

In Epoch 5

… the training loss and perplexity continues to plunge (loss  4.39 | ppl    80.49), but as usual this results in no gain in the validation loss and perplexity. In fact things are getting even worse, far worse (valid loss 6.5190 | valid ppl   677.93), — dismal values. Perhaps this relates to how the Corpus is split, maybe the validation is all from a different website. PyTorch does not seem to allow in this word-language-model a shuffling and randomized cutting of the corpus.

The epoch 5 model seems to be a medievalist, then a modernist, ironic perverse dimwitted saucy in turns:

Epoch: 5
Loss: 6.52


 Him Saviour drives his heavy wine; 
 full fraught with brass, they draiks, 
 cell-sap mimic ease in bitch? 
 The living bird of England flies, 
 And found a slowly sigh, and spred 
 The coming of a thousand feeling, 
 And all the trouble and his graves! 



 And trust the pain of life and feeling, 
 And cows in praise and beguile. 
 heh will have ten or three things reach. 


we saw a look in the river of sleep. 
 Summer in winter 
 winter light flies 
 over noon!


 Impenetrable, mouth 
 stretched on by light. blotted ice 
 through the sprawling roof, the switch 
 made plate to eternities. ovaries. 
 Last night my window flattened through 
 a veil of Magnin's. light cot, 
 Precedence phlox with graced, 
 ice horse long telephone Japanese passbook 
 under the envelopes of travelers 
 pushed up

    ~ + ~


 Though we dont visit ourselves 
 as if they ever looked up like anyone 
 in one nowhere of existence 
 with passionate eyes that dreamed 
 where they ] in their slow bodies 
 like crystal sods on a black marketplace.

Epoch 6

The learning rate is now descending rapidly. It gets divided by 4 with each epoch: 20, 5, 1.25, and now it’s 0.31. So it’s almost time for me to goto pilates. To escape the long slow tedious sprawl of math evolving beneath the metallic skin of my cooler master computer housing.

Even as the testing loss and perplexity descend to perplexing levels (loss 4.24 | ppl 69.55), and the validation soars to new untenable ludicrous values (valid loss 6.5382 | valid ppl   691.04), generation continues to produce spasms of primordial elegy:

Epoch: 6
Loss: 6.54


 And their minds are mouths of no heads. 
 In the woods the gulls are not built 
 On the glowing mouth of the door; 
 Where the guards go, as they written. 
 The woodbine comes slowly fill'd 
 Beyond the shadow of the roads, 
 Wandering where silence is born


Houses of trees, of shadow, autumn's bows; 
 I am unable to do the god and I see. 



 The red and feathers of the autumn lime, 
 The baskets of honey, the living breath 
 And the blind face of an windpipe it. 
 But the death who leaves the silent god and dust


 One more mad than death. 
 Perhaps there were difficult doubting the sea 
 Shooting this language. 
 I am the Smoke talking 
 It was worth thought.



 until the half-page hours grind the horizon, 
 a swooping harvest toward a sheet, its fingers 
 a vapoury tinny sky.

Epoch 7

Testing (loss  4.21 | ppl    67.03) went up a bit. Validation almost the same. Poetry continues even as the poet’s change.

Epoch: 7
Loss: 6.53

 That evening, when the darling, 
 fell out on a redwood P.A. hut, 
 rain fell like a dinosaur exultances, blue, 
 cool, pebbles, greenhorn, while modify 
 to the shed, his eyelids 
 actual in the glass. 

 ~ + ~


 This is the story of the people. 
 But just as he changed at first the man, 
 Ear, a grimy Jupiter's lawn 
 abides, and cheered itself 
 The world could be his last.


With the narrow loves of a jumper 
 Driving into the sun, 
 Round the rocks 
 Like the singular 

Epoch 8

Nothing can be said to appeases such blatant non-linearity:

Epoch: 8
Loss: 6.53

 Words and true deadweight 
 more than what I want is to rise 
 in earth in storms of mythological, 
 love, and less without old Limping 

~ + ~


 Law on the wall I was used to put up 
 In the ocean's coffee.
 but where people did gap at Destinie, 
 close to the face 
 where they waited 
 their deep broken patches 
 once in the hollow water 
 of motion word glimmering 
 inside their fiery scale 
 but we all carried ourselves 
 in all our happiness 
 There, quavering. 
 the night we descend for drummed, 
 colonial secular animals 
 bled to maim'd glass. 

 ~ + ~

 Was the last cold wind of hell 
 That day long, two days sea-cold be, 
 In grass that sounds and remorse.

… & after 16 Epochs…

Perhaps due to the duplicitous influence of my expectations, or maybe due to the beneficial endorphin vibe induced by returning from pilates along snow-strewn sidewalks, but does it not seem that the primary cohesion of these verses is a micron stronger? Yes, however these were selected from amongst the drifting warp of nonsense:

Epoch: 16
Loss: 6.50


 collecting disheveled, flowers, 
 Rising courses, wood, lean plums 
 Where snakes burn down from airs 
 That swelled his bleeding burden 
 afflatus gathering Banks, 
 On the lemon cloud soft as wet 
 The curtains signals the sky. . . . 

 So that, it is so like that. 
 Everything is strong. 
 The water echoes and river-drifts 
 of leaves and straw. 
 On the bare horizon the border 
 heaves over the perch, is 
 what I would. 


No answer. No shrine, no leaders of expulsion or yearning more 
 Nor than an ideal planet 

The Flat-Hunters Answers to Pasadena 
 Neither our soft Name Incredible 
 Out of itself, the flat white bear 
 Above the dawn. And now between us 
 A dead man's life will not, 

 ~ + ~


 See how the wild autumn towers and Summer springs, 
 Where the solitary yellow swallow its wheel 
 But the foothold of the mist, 
 Fierce and thick, indifferent 
 On the edge of the Releas'd, a warning 
 An imitation of mastodon 
 Whose light is swallow, and creeping is to be pride, 
 Look! Lucid passes back and again

& last Epoch, Epoch 17

With learning rate at zero even incremental change has ceased. We are now down to vibratory whimpers of variational deviances.

Epoch: 17
Loss: 6.50


 I worshipped myself in a dream-- 
 The man watching his possibility and pain -- 
 Said he said of my little story. 
 Then he told him he'd an untoward car. 
 Then he had killed my animal's wished for hell 


Walks through the perishable pall as she walks down 
 Her bargains and her guardians of the soft 
 That is no face. 
 She thrusts of her corruption and her favorite meals. 
 My beautiful Girl in the thick face he sang in her hand. 
 Praise her hair, tell me the tell of the joy,-- up; 
 I'd stayed for her terror and her skinny heart 
 And fate's shadowy looking at an acorn 
 Pushed at the tired, and dressed in quiet


We fall upon a glowing immaculate seat


 The fact, turns home to the thick of the waters 
 Embraces the surf must go on: all 
 When our borders were ash arm. 
 We could live on dead. revolution, 
 flowed down from the floor to be seen to come to rammer, 
 Drunk be untranslatable in us.


The terrible bath, 
 The grasses and the debris 

Complete text of test runs: here

Code: here

Video: here

The GTX TitanX GPU used in this research was generously donated by the NVIDIA Corporation.

, , , ,