Category: data art

PyTorch Poetry Generation [Pre-WordHack : Epoch 16 Video]

Another day of testing before going to NYC to perform neural-net poems at WordHack [NYC (Thursday 2/16/2017 @ Babycastles . 7-10pm) w. Sarah Rothberg, John Cayley and Theadora Walsh]

HOPE 

 In the cold weather going out of the snow, 
 She down the lawn. 
 
 The air moves and grows, while she walks smooth, 
 When a swan is born, 
 And it's almost happening 
 
 Who knows what to say 
 The change has brought 
 Throwing the first blood in its face.

It’s clear:

Never will this mode of randomized pattern-reasoning replicate the nuanced human heart. More robust ensemble methods that simulate embodied experience, temporal reflexes, and nested community idioms will be required.

Deep learning is still shallow. The cloud does not understand honey, home or heart. Yet in the short-term, this is the future of writing: a computational assistant for an engaged imagination intent on exploring the topological feature-space of potential phrases.

Done:

Modulated the parameters: raised both embedding size and hidden layers to 512. And did a bit more data mining and parsing to increase the corpus size by 1/3 to 20mb of .txt.

Mode: LSTM
Embedding size: 512
Hidden Layers: 512
Batch size: 20

Expanded Corpus to over 600,000 lines

639,813 lines of poetry from 5 websites.

Poetry Foundation
Jacket2
Capa - Contemporary American Poetry Archive
Evergreen Review
Shampoo Poetry

Continue reading

Testing PyTorch on Poems (Preliminary Results)

PyTorch is an early release beta software (developed by a consortium led by Facebook and NIVIDIA), a “deep learning software that puts Python first.”

So since I luckily received an NVIDIA GTX TitanX (Maxwell) before leaving Hong Kong under the generous NVIDIA academic GPU Grant program, and having last week finally bought a custom-build to house it, and 2 days ago finally got Ubuntu installed with CUDA and CUDNN drivers, and having found that the Tensorflow 0.11 version no longer runs under Python 3.6 Anaconda, I decided to give a PyTorch example a try, specifically Word-level language modeling RNN

This example trains a multi-layer RNN (Elman, GRU, or LSTM) on a language modeling task…The trained model can then be used by the generate script to generate new text.

And after only an hour of training on an 11k poem corpus, using the default settings, the results announced “End of training | test loss  5.99 | test ppl   398.41” — Which means that the loss is bad and perplexity is now at the seemingly terrible level of 398….

Then I ran the generate script and the 1000 word text below got generated in less than 30 seconds. I find it stunning. If this is what PyTorch is capable of with a tiny corpus, default settings and a minimal run, language generation is entering a renaissance.  Ok, so it’s veering toward the incomprehensible and has little lived evocative phenomenological resonance, but its grasp on idiomatic cadence is creepily accurate. It’s as if it absorbed several semesters of graduate seminars on romantic and post-modern verse:

the embankment
and your face sad like a nest, grew sorry
when your cold work made of snow
broken
and left a thousand magnifies.

a little cold, you plant but hold it
and seems
the slight arts? face, and ends
with such prayer as the fingers do,
this reorganizing contest is how
to be murdered
throwing it
into the arteries obscurity goes disc whispering whole
affairs, now your instinct
does a case,
defense. on her eye, you do not know that every homelands
is didn’t at the
risk very psychiatrists, just under bay.

by the living of life’s melancholy grate.
i have found a
wild orange in eden, eight hazy years guzzles
her neck at the grave turn into every mythological orbit of
distances,
person’s there–see then are we told what we understand
won’t take the slightest danger
or the
size of what it means to take up if you can,
tongue. only your eye exultant whitens again will
happen.
i think that the four-oared clouded of one stick in flowerpot
is part of an antique little
register on a hiatus
till i try for you.
i wash up the door my knee will be
high.
if i refuse a limits as i can lift my hand rubicon.

i can see her
above the stove tide
hip. orange as a breaking sty.

Continue reading

T-SNE Animator

Same Data, Same Code (Different Parameters)

Jhave (2016)
Python code, T-SNE algorithm, 6689 poems
Cloud support by Karteek Addanki

Information visualizations normally change as the data changes. In this demo, the data (6689 poems) stays the same, but visualizations change as the parameters change.

HD video generated from Python script.
Project Code on Github: https://github.com/jhave/TSNE-animator
Exhibited as part of pop-up exhibit for Digital Asia Hub forum on AI in Asia (21/11/2016) at Maritime Museum, Hong Kong

 

Wavenet for Poem Generation: preliminary results

For the past week, I’ve been running a port of the Wavenet algorithm to generate poems. A reasonable training result emerges in about 24 hours, — a trained model that can generate immense amounts of text relatively quickly. On a laptop. (Code: github). By reasonable I mean the poems do not have any real sense, no sentient self, no coherent narrative, nor epic structure. But they do have cadence, they do not repeat, new words are plausible, and they have adopted a scattered open line style characteristic of the late twentieth century corpus on which they were trained. Much more lucid than Schwitters’ Ursonate, output is reminiscent of Beckett’s Not I : ranting incandescent perpetual voice.


Results

Remember, these are evolutionary amoebas, toddlers just learning to babble. The amazing thing is that without being given any syntax rules, they are speaking, generating a kind of prototypical glossolalia poem, character by character. Note: models are like wines, idiosyncratic reservoirs, the output of each has a distinct taste, — some have mastered open lines, others mutter densely, many mangle words to make neologisms  — each has obsessions. The Wavenet algorithm is analogous to a winery: its processes ensure that all of the models are similar. Tensorflow is the local region; recursive neural nets form the ecosystem. The corpus is the grapes.

Intriguing vintages-models :

Dense intricate Model 33380 — trained with 1024 skip channels and dilation to 1024 (read a txt sample)

the mouth’s fruit
tiny from carrying
a generative cup

Loose uncalibrated Model 13483 with loss = 0.456, (1.436 sec/step) trained on 2016-10-15T20-46-39 with 2048 skip channels and dilation to 256 (read a txt sample)

 at night, say, that direction.

          sleeps
      now. so you hear we are shaking
          from the woods

Full results (raw output, unedited txt files from the week of Oct 10-16th 2016) here.

it’s there we brail,
  beautiful full
left to wish our autumn was floor

Edited micro poems

…extracted from the debris are here.

through lust,
and uptight winking cold
blood tree hairs
   burned
 in loss

Code

Python source code + a few trained models, corpus and some sample txt: on github which will be updated with new samples and code as it emerges.


 Details

The Model number refers to how many steps it trained for. Skip channels weave material from different contexts. On this corpus, larger skip channels produce more coherent output. Dilations refer to the size of the tensors of the encoder-decoder: eg. [ 1, 2, 4, 8, 16, 32, 64, 128, 256, etc… ] Higher values up to 1024 seem to be of benefit, but take longer to train. Loss is the mathematical calculation of the distance between the goal and the model; it is a measure of how tightly the model fits the topological shape of the corpus; as models are trained, they are supposed to learn to minimize loss; low loss is supposed to be good. For artistic purposes this is questionable (I describe why in see next section). For best results, in general, on this corpus: 10k to 50k steps, 1024 dilations, a skip channel of 512 or more, and (most crucial) loss less than 0.6.


Learned

Loss is not everything. An early iteration model with low loss will generate cruft with immense spelling errors. Thousands of runs later, a model with the same loss value will usually produce more sophisticated variations, less errors. So there is more going on inside the system than is captured by the simple metric of loss optimization. Moreover if the system is about to  undergo a catastrophic blowout of loss values, during which the loss ceases to descend toward the gradient and exponentially oscillates (this occasionally occurs after approx 60k steps). Generated text from poems just before that (with good loss values below 1.0 or even excellent loss values below 0.6) will produce some ok stuff interspersed with long periods of nonsense or —— repeated **** symbols. These repetitive stretches are symptoms of the imminent collapse. So loss is not everything. Nonsense can be a muse. Mutating small elements, editing, flowing, falling across the suggestive force of words in raw tumult provides a viable medium for finding voice. Continue reading

LSTM CHARRNN: blossoming acronyms

Machine learning hacks.
Building poetic nonsense with neural nets.

 

mules and the technology, the created and the tractions and the tractional artically of the traction of the tractical processe of the prectional and and and structured the entional the eractions of the the tractions and the tractions of the termore the the creative of the ention of the metach of the interallention of the termore and the entions of the created the tractions and structured the media and contempler the tractically, and in the eranted the tractions of the metach of the contempler in

 

The network knows nothing of language.
Sense emerges from feedback.

 

Using Theano backend.
Source text on which the neural net is trained:  MITPressDRAFT

 

! ” # % & ‘ ( ) + , – . / 0 1 2 3 4 5 6 7 8 9 : ; = ? @ [ ] ` a b c d e f g h i j k l m n o p q r s t u v w x y z

 


After cleanup total chars: 61
Corpus length: 421183

 

My days evolve:
Making micro mods to Keras code.
Reading up on LSTM.
Watching Sirajology.

 

As usual, there is tons of text to read…

 
Continue reading

ELO 2015 — Bergen – Performance

One of the ends
of digital literature
is an external intuition.

External intuition
is an engineering problem.

Intuition
in this case
is me.

Skidding thru the
generated poems
as they augment
my imagination.

I call this act of
augmented imagination:
cyborg/ skid/ spreedr poetry.

####

For the performance,
at ELO conference performance in Bergen, Norway on Aug. 5th 2015
I generated 100 poems, pausing ever 5 poems for 20 seconds,
and tried to weave spoken poetry from the output.

Unfortunately I forgot to record the audio/screenvideo
so my actual spoken-poem is gone!

#####

If you want to try it yourself:
Download attached code and data corpus
— this includes the poems generated during the performance.

Unzip
run in command line
>>cd code/poetryFoundation/ELO_July2015/
>>python ELO2015_PERF_Creeley-Aug4th.py

This code will run with Python package Anaconda 2.7 installed.

####

Code will continue to be updated on
on github: https://github.com/jhave/Big-Data-Poetry.

Spreeder: the feature film (EPC 20th Anniversary Celebration)

Loss Pequeño Glazier is celebrating the 20th anniversary of the Electronic Poetry Centre along with Charles Bernstein, cris cheek, Tony Conrad, Steve McCaffery, Myung Mi Kim, Tammy McGovern, Joan Retallack, Laura Shackelford, Danny Snelson, Dennis Tedlock, Cecilia Vicuña, Elizabeth Willis, & Wooden Cities with Ethan Hayden. Along with exhibitions by: “Abra” (Amaranth Borsuk, Kate Durbin & Ian Hatcher); Pry: iPad-based Novella (Samantha Gorman & Danny Cannizzaro); “Enter:in’ Wodies” (Zuzana Husárová & Lubomír Panák) & myself.  http://epc.buffalo.edu/e-poetry/2014/EPC-at-20/

So I made a future-feature film of a computer writing in real time
Spreeder (approx. output 8,100 poems; 2 hour-long real-time episodes).
Implemented in Python with Alchemy API, NLTK and pattern.en

 

 

SPREED : Speed Screen Reading : One Hour Real-Time Poetry Generation ScreenGrab

Using Python (Anaconda), NLTK, WordNet, Alchemy, pattern.en, and pyenchant
to analyze and perform word replacement
on a corpus of 10,119 poems scraped from the PoetryFoundation
and generate 7,769 poems in approx. 2 hours and 30 minutes.

This is a real-time hour-long screen-grab output
of the trace window in SublimeText
as the poetry-gen program runs.



Code on Github
Made by Glia.ca  


And here is another episode of  “Crawling Toward Creeley” (in this episode: a variation on the ‘Gnomic Verses’ generated then modified)

Markov Bern

Markov chains are one of the traditional tricks in the NLP playbook. They are the apple pie-chart of text-generation.

Basic process: given a source text, find words that are neighbours, if you know the neighbours of a word, you can form a chain if you wish. [(“you”),(“know”,”can”,”wish”)] and reconstruct a text which contains pairs (bigrams) from the source.

So I did that using as source texts: Charles Bernstein Dark City and Rough Trades. (Found on Bernstein’s EPC author page).

The result is an example of what Charles Hartman might refer to as newbie-augmented-cyborg-poet (dead simple technically, but satisfying artistically since the volume of generated texts from which new verses can be hand-crafted is massive). This sort of auto-suggest based-upon-a-corpus technique radically shifts the dimensions of creativity: in the first ‘modified’ example I edited the output, adding words, disguising some obvious quotations from Bernstein, truncating verses, changing lines, modulating rhythms. In the raw output below, it’s just the computer (fueled by Bernstein’s berning phrases), it could go on infinitely given a large enough corpus.



Poetry is both the easiest and the hardest to generate. Since non-linear deflections and word-riffs are an aspect of contemporary poetry, slamming together ripe fertile conjunctions is easy. Migrating toward a sensitive, complex, experiential and contextual lived poetry is the real challenge (I didn’t even begin to touch it here).


Code on Github
Made by Glia.ca  


what is there if
An exit is as
clear as dead.

Billboards poster our losses.
Better a barber
than a thousand one-line
sweat glands.

SYN-SCI-RAP

I think I have begun to develop a mild form of insanity that often strikes those who fiddle around with computationally-generated text. After reading thousands of lines of dense incomprehensible gibberish it clarifies and makes sense, often more sense than any mere linear thought. The brain acclimatises to syntactic pressure.


Recipe for mildly insane word-salad:

  • take 57,000 rap songs input by fans,
    • extract all words that do not return results from WordNet synset search and put into Reservoir
  • one list of scientific terminology (for sombre intellectual tone)
    • chop off “-ology” wherever it occurs
  • one list of swear words (for spice)
  • call to WordNet synset algorithm (for fibre and continuity)
  • use pattern.en to do conjugation (for a tiny bit of coherence)
  • use NLTK part-of-speech tagging
  • Alchemy for entity (people, places, etc…) replacement
  • 10,000 or more poems

Mix all ingredients together using replacement algorithms.


To read 10116 poems (simple style) (in a single 24-mb html page) generated in 10356.4216051 seconds (2.87 hours, 3612pph [poems per hour], 60 ppm [poems per minute] ) on 2014-08-14 at 02:54 click here


Read a selection of just a few poems 

Read the RAP Reservoir: 33,150 words extracted from 56k user-input rap songs that did not return any usable results from a WordNet synset search. If you are looking for the evolution of language that occurs through mutation (typo, mispells, pop-cruft) this is it.


Code on Github
Made by Glia.ca