I've been training a neural network on the stroke progressions of handwriting paths, which I previously coded using JavaScript & p5.js. I'm using Graves' LSTMs, a sequence model that generates handwriting as paths, implemented in PyTorch.
While my original system always writes real words, the network invents marks and glyphs, creating asemic writing which feels like the same hand, but drifts into half-words and meaningless symbols.
Training material
I designed nine different handwriting styles to use as training material, by adjusting the parameters in my p5.js handwriting system.
The model trains on the raw path data, formatted in a JSON file. I generated paths for 300 sentences for each of the 9 styles.
I built a viewer which parses the JSON files so that I could view the training data visually and, later, view the generated paths.
First round of training
In this initial round of training Graves' model I used the raw path data of the 2700 sentences (300 each for 9 styles) and trained for 40 epochs.
As training was running, I began using the model to generate new paths and, by comparing the results from epoch 6 to epoch 40, it's clear to see how the model improves.
By epoch 40 we're starting to see recognisable letter forms scattered around, but there are no readable words.
Developing training material
I wanted to achieve results that look closer to the original paths so I decided to try training on just one style of handwriting at a time to give the model an easier task.
The paths in my original code contain many points which are very close together, due to the way they're created with Chaikin's curve algorithm. I resampled these paths to create more even spacing, so the model could train on more meaningful jumps from point to point.
Many of my original different text styles are created by essentially stretching the letters in different directions. This seemed to have an interesting result on the generated paths, where the model became unable to keep the 'sentences' in a straight line. I believe this is due to the fact that the model has learnt that the paths move vertically a lot, but hasn't learnt that they usually do about the same amount of movement up as down.
Text-conditioned model
All the training so far had been done with Graves' prediction network, which is designed for handwriting without text information. At this point I switched to Graves' synthesis model, which also takes the sentences being written as text strings, enabling it to attempt to learn the specific letter paths.
It uses a "soft window" to deduce where letters begin and end, so called because, instead of creating hard boundaries around letters, it increases weighting of letters as the window slides forward through the text.
I also increased the number of training epochs to 80. This training took longer for each epoch but, as the epochs progressed, meaningful sentences began to appear. This model also enables the definition of a specific sentence in the output. I began with a sentence directly from the training material.
I then tried outputting a sentence not in the training material, "Amazingly few discotheques provide jukeboxes", which uses every letter of the alphabet. I was pleasantly surprised that it handled this reasonably well too.
Next I tried adjusting the 'temperature' parameter, which controls how close the model stays to the more likely path progressions or how much it strays, which produced some interesting results. The image above is generated using temperature 0.4.
Letter-labelled paths
I've also explored using a network that takes a label on each point in the paths, defining which letter it belongs to. This produces results that are not as close to the original handwriting, I think this must be because the "joined up" handwriting lends itself well to the soft window approach. However, I enjoy the mysterious scribble effect.
On the left is a variety of attempts at writing "I deconstruct feelings" taken from different epochs and different training sets. On the right it's the phrase that uses every letter of the alphabet again, where it becomes evident that there was some problem with j's in the training set.
Next I used the letter tagging method and trained on a large set of the wavy handwriting style, to generate some more gestural results.
These keep the rhythm of handwriting while sitting between readable and asemic. I'm interested developing both the methods that produce readable versions of any given sentence, and the ones that drift into asemic writing and even scribbles.
A next step would be to try a transformer on the same paths, which should handle longer sequences more coherently. I'd also like to experiment with mixing the handwriting styles in different ways, and explore where I can use these generated paths in my work.