Another day of testing before going to NYC to perform neural-net poems at WordHack [NYC (Thursday 2/16/2017 @ Babycastles . 7-10pm) w. Sarah Rothberg, John Cayley and Theadora Walsh]
HOPE In the cold weather going out of the snow, She down the lawn. The air moves and grows, while she walks smooth, When a swan is born, And it's almost happening Who knows what to say The change has brought Throwing the first blood in its face.
It’s clear:
Never will this mode of randomized pattern-reasoning replicate the nuanced human heart. More robust ensemble methods that simulate embodied experience, temporal reflexes, and nested community idioms will be required.
Deep learning is still shallow. The cloud does not understand honey, home or heart. Yet in the short-term, this is the future of writing: a computational assistant for an engaged imagination intent on exploring the topological feature-space of potential phrases.
Done:
Modulated the parameters: raised both embedding size and hidden layers to 512. And did a bit more data mining and parsing to increase the corpus size by 1/3 to 20mb of .txt.
Mode: LSTM Embedding size: 512 Hidden Layers: 512 Batch size: 20
Expanded Corpus to over 600,000 lines
639,813 lines of poetry from 5 websites.
Poetry Foundation Jacket2 Capa - Contemporary American Poetry Archive Evergreen Review Shampoo Poetry
After only 1 epoch
… of training (basically the system has read the corpus once and created a model based upon that reading, a process that takes about 12 minutes on TitanX), the system spits out semi-plausible dada-juice:
Epoch: 1 Loss: 6.38 Perplexity: 582.92.pt THIN-ROOTED, PURITY; ARCHAEOLOGIST DOWN RE-WRAP The bones of my Chasms Round the eye with day what hair Is in the waves that wrote us To a black blade, of vowel fed.
After 2 Epochs
… the juice gets sporadically slightly thicker:
Epoch: 2 Loss: 6.27 Perplexity: 526.68.pt My feet grow lattices where their horses rise to their lips. The girls go Vanish, up & left twenty murmur'd go out of my imaginary
By Epoch 3
… things continue to unravel resiliently, even as the validation loss increases slightly, the testing loss is descending, which may of course signal the dreaded overfitting, — but this being poetry and not finance, i do not care as long as the system produces lines which never existed before.
In this instance, overfitting may mean that the model is becoming capable of speaking in any style or hybrid mashup glomp :
Epoch: 3 Loss: 6.36 Perplexity: 577.60.pt As A yellow brown darkens Must come, no shape Than low undone, Beside the water enraptured-- hungry; of youngish found The couch'd fox-head within its harp, Came from the earth uncivil; So the moss were mixed And bent off
At Epoch 4
…validation loss and perplexity are even worse! But the test loss and perplexity (loss 4.83 | ppl 125.05 ) are way better. Let’s dive into that sculpted overfit snug wash of poetry, it seems as if the model has become besotted and fallen in love with love:
Epoch: 4 Loss: 6.42 Perplexity: 614.91.pt I am the torch a letter is lov'd out, with a white saint scrolls from one hand in the driveway. My blazing soul is hungry the tongue can tell the brusque King of the Lord Oldsmobile ... I saw as proclaimed: with his drape. I put a straight mine for my sunflower She moved; the knee and slow his foot Shows to each stroke of things
In Epoch 5
… the training loss and perplexity continues to plunge (loss 4.39 | ppl 80.49), but as usual this results in no gain in the validation loss and perplexity. In fact things are getting even worse, far worse (valid loss 6.5190 | valid ppl 677.93), — dismal values. Perhaps this relates to how the Corpus is split, maybe the validation is all from a different website. PyTorch does not seem to allow in this word-language-model a shuffling and randomized cutting of the corpus.
The epoch 5 model seems to be a medievalist, then a modernist, ironic perverse dimwitted saucy in turns:
Epoch: 5 Loss: 6.52 Perplexity: 677.93.pt PAVEMENT, LOVE HAS PUBERTY. Him Saviour drives his heavy wine; full fraught with brass, they draiks, cell-sap mimic ease in bitch? The living bird of England flies, And found a slowly sigh, and spred The coming of a thousand feeling, And all the trouble and his graves! ... SEEKS, And trust the pain of life and feeling, And cows in praise and beguile. heh will have ten or three things reach. ... we saw a look in the river of sleep. Summer in winter winter light flies over noon! ... Impenetrable, mouth stretched on by light. blotted ice through the sprawling roof, the switch made plate to eternities. ovaries. Last night my window flattened through a veil of Magnin's. light cot, Precedence phlox with graced, ice horse long telephone Japanese passbook under the envelopes of travelers pushed up ~ + ~ LEFT TO LAST. Though we dont visit ourselves as if they ever looked up like anyone in one nowhere of existence with passionate eyes that dreamed where they ] in their slow bodies like crystal sods on a black marketplace.
Epoch 6
The learning rate is now descending rapidly. It gets divided by 4 with each epoch: 20, 5, 1.25, and now it’s 0.31. So it’s almost time for me to goto pilates. To escape the long slow tedious sprawl of math evolving beneath the metallic skin of my cooler master computer housing.
Even as the testing loss and perplexity descend to perplexing levels (loss 4.24 | ppl 69.55), and the validation soars to new untenable ludicrous values (valid loss 6.5382 | valid ppl 691.04), generation continues to produce spasms of primordial elegy:
Epoch: 6 Loss: 6.54 Perplexity: 691.04.pt THROUGH THE HILLS WHERE WAVES ARE EXTINGUISHED, And their minds are mouths of no heads. In the woods the gulls are not built On the glowing mouth of the door; Where the guards go, as they written. The woodbine comes slowly fill'd Beyond the shadow of the roads, Wandering where silence is born ... Houses of trees, of shadow, autumn's bows; I am unable to do the god and I see. ... THE FOLIAGE SLANT AT SUNSET. The red and feathers of the autumn lime, The baskets of honey, the living breath And the blind face of an windpipe it. But the death who leaves the silent god and dust ... One more mad than death. Perhaps there were difficult doubting the sea Shooting this language. I am the Smoke talking It was worth thought. ... AGAIN, SHACK until the half-page hours grind the horizon, a swooping harvest toward a sheet, its fingers a vapoury tinny sky.
Epoch 7
Testing (loss 4.21 | ppl 67.03) went up a bit. Validation almost the same. Poetry continues even as the poet’s change.
Epoch: 7 Loss: 6.53 Perplexity: 686.91.pt That evening, when the darling, fell out on a redwood P.A. hut, rain fell like a dinosaur exultances, blue, cool, pebbles, greenhorn, while modify to the shed, his eyelids actual in the glass. ~ + ~ OUTCROPS-- IN LABYRINTH AND PUNCTURED. This is the story of the people. But just as he changed at first the man, Ear, a grimy Jupiter's lawn abides, and cheered itself The world could be his last. ... With the narrow loves of a jumper Driving into the sun, Round the rocks Like the singular Souls
Epoch 8
Nothing can be said to appeases such blatant non-linearity:
Epoch: 8 Loss: 6.53 Perplexity: 688.07.pt Words and true deadweight more than what I want is to rise in earth in storms of mythological, love, and less without old Limping ~ + ~ THE STRANGE Law on the wall I was used to put up In the ocean's coffee. * but where people did gap at Destinie, close to the face where they waited their deep broken patches once in the hollow water of motion word glimmering inside their fiery scale but we all carried ourselves in all our happiness There, quavering. the night we descend for drummed, colonial secular animals bled to maim'd glass. ~ + ~ MISSING. Was the last cold wind of hell That day long, two days sea-cold be, In grass that sounds and remorse.
… & after 16 Epochs…
Perhaps due to the duplicitous influence of my expectations, or maybe due to the beneficial endorphin vibe induced by returning from pilates along snow-strewn sidewalks, but does it not seem that the primary cohesion of these verses is a micron stronger? Yes, however these were selected from amongst the drifting warp of nonsense:
Epoch: 16 Loss: 6.50 Perplexity: 663.70.pt THEN I ROSE collecting disheveled, flowers, Rising courses, wood, lean plums Where snakes burn down from airs That swelled his bleeding burden afflatus gathering Banks, On the lemon cloud soft as wet The curtains signals the sky. . . . 4 So that, it is so like that. Everything is strong. The water echoes and river-drifts of leaves and straw. On the bare horizon the border heaves over the perch, is what I would. ... No answer. No shrine, no leaders of expulsion or yearning more Nor than an ideal planet ... The Flat-Hunters Answers to Pasadena Neither our soft Name Incredible Out of itself, the flat white bear Above the dawn. And now between us A dead man's life will not, ~ + ~ SWIMMIN'-HOLE. See how the wild autumn towers and Summer springs, Where the solitary yellow swallow its wheel 1987 But the foothold of the mist, Fierce and thick, indifferent On the edge of the Releas'd, a warning An imitation of mastodon Whose light is swallow, and creeping is to be pride, Look! Lucid passes back and again
& last Epoch, Epoch 17
With learning rate at zero even incremental change has ceased. We are now down to vibratory whimpers of variational deviances.
Epoch: 17 Loss: 6.50 Perplexity: 663.43.pt ACCOMPLISHED SORTS) I worshipped myself in a dream-- The man watching his possibility and pain -- Said he said of my little story. Then he told him he'd an untoward car. Then he had killed my animal's wished for hell ... Walks through the perishable pall as she walks down Her bargains and her guardians of the soft That is no face. She thrusts of her corruption and her favorite meals. My beautiful Girl in the thick face he sang in her hand. Praise her hair, tell me the tell of the joy,-- up; I'd stayed for her terror and her skinny heart And fate's shadowy looking at an acorn Pushed at the tired, and dressed in quiet ... We fall upon a glowing immaculate seat ... The fact, turns home to the thick of the waters Embraces the surf must go on: all When our borders were ash arm. We could live on dead. revolution, flowed down from the floor to be seen to come to rammer, Drunk be untranslatable in us. ... The terrible bath, The grasses and the debris enrichment
Complete text of test runs: here
Code: here
Video: here
The GTX TitanX GPU used in this research was generously donated by the NVIDIA Corporation.