Challenges to Comprehension Implied by the Logo
of Laetus in Praesens
Laetus in Praesens Alternative view of segmented documents via Kairos

8th July 2007 | Draft

Convertor from Text to Poetry, Song or Music

Computer-assisted aesthetic enhancement of treaties, declarations and agreements

-- / --


Introduction
Preamble: Aesthetic | Fidelity
Distinguishing modes of aesthetic sonification
Conventional aesthetic enhancement and adaptation of constitutive documents
Interrelated modules, challenges and applications
-- Text focus | Speech focus | Semantic focus | Text and poetry generators
-- Content augmentation | Sound/Auditory focus | Image focus
-- Poetic transformation of text | Setting to music/song | Musical rendering
Related initiatives
Conversion options (steps)
Possible texts for experiment
References

Introduction

The medium on which constitutive psychosocial patterns (legal texts) are articulated, or through which they are expressed, effectively continue to follow the ancient tradition of inscribing such directives or edicts "in stone".

The current evolution of software techniques for text parsing notably allows for its conversion into speech in text readers (for those disinclined to read text displayed on a screen). More complex translation software allows text to be converted into text in another language, possibly also output as speech. Voice recognition software adds to such possibilities. Experiments have long been undertaken with computer generation of poetry (one variant of digital poetry) and of music from seed elements. Music is itself commonly used to trigger dynamic visual patterns.

The concern here is with the computer-assisted systematic conversion of text into poetry, song or music. The processes described focus on the desirable options, the specific software challenges, and the useful applications associated with various stages of development -- irrespective of early fulfillment of all envisaged possibilities. The emphasis is on the "aesthetic enhancement" of texts of fundamental treaties, declarations and agreements. The object is to render the semantic content of such texts more widely comprehensible and memorable, without losing their essential significance, notably for the benefit of those who prefer non-textual communication modes. Many other applications may be envisaged, perhaps including inter-faith and pre-nuptial agreements.

An important reason for such conversion is that text processing is primarily a function of the left hemisphere of the brain whereas other modes vital to comprehension are a fiunction of the right hemisphere. The conversion is therefore specifically designed as a means of transferring significance from one to the other as a means of benefitting from the particular integrative skills of the second. This facility is potentially of value as a feedback mechanism when endeavouring to articulate treaty-type texts. These depend in some measure on the manner in which their significance can be effectively communicated to people who may have little inclination to read texts -- even if they are not functionally illiterate. The facility is also intended as a means of offering various forms of summary or synthesis with ready appeal to the media.

It could be argued that legal texts, as currently conceived, represent what might be described as the "lowest common aesthetic denominator". It is not surprising that many are apathetic about them and are more highly engaged by other styles of presentation.

The essential practicality of the approach here is that it avoids the major problems of voice recognition and text translation, although these are now increasingly soluble. Nor is it directly concerned with creative generation of text, poetry or song. The challenge is constrained to enhancement of comprehension through introduction of aesthetic qualities into the existing semantic patterns provided by input text.

Preamble

Aesthetic: As noted by Gerald England (Poetry and Computers. Fractal Report, 6, 1994, 31)

There have been attempts to produce computer-generated poetry for over 30 years - most of these have been done to investigate problems in programming rather than been serious literary endeavours. There are programs about that purport to generate poetry. Most rely on a pre-determined set of adjectives, nouns, verbs and adverbs randomly applied. The results are pretty dreadful! Fact is computers can't write poetry, only poets can.

However the challenge is whether existing (rather than computer-generated) prose can be manipulated ("massaged") by computer algorithms under user-supplied constraint in order to increase, using various devices, the aesthetic order (however this is to be understood) of the prose. England himself describes his experiments with a text generator programme called Babble.

Computer-based initiatives to produce poetry are associated with technical challenges that are of some interest in relation to solution of a range of problems with more obvious applications.

It is appropriate to note, for example, the interrelationships of the initiatives of inventor Raymond Kurzweil (pioneer in optical character recognition (OCR), text-to-speech synthesis, speech recognition): Kurzweil Educational Systems develops reading-related technology for the academically challenged; KurzweilAI.net explores "the confluence of accelerating revolutions that are shaping our future world". Kurzweil CyberArt Technologies offers the (patented) Poet's Assistant, the Poet Analyzer, the Poet Creator -- that help write poetry (and song lyrics). [These have not been adapted since 2000 to the latest operating system environments]

As reported by Jordan Boyd-Graber (Semantic Poetry Creation Using Lexicographic and Natural Language Texts, 2000), in a well-controlled test, the Kurzweil Cybenetic Poet managed to fool about half the people half the time. People with moderate computer and poetry experience usually fared the best, only being fooled 5% of the time. However, the overall average was about 44.5%. This is still better than blind luck, but the program does sometimes produce frighteningly realistic responses.

Whilst the following exploration does not preclude the generation of intriguing products that are challenging to simpler aesthetic preferences, the focus is on output that enhances comprehensibility for larger numbers of people of a wider variety of backgrounds.

Fidelity: As a Christian evangelical initiative the Jubilate Group provides songs and hymns for worship. The group has concerns which apply in some measure to the challenge of setting authoritiative international treaties to poetry or music:

We could set a bible text to music, faithfully reproducing the text, but with a few exceptions people find this unsatisfactory. This is in much the same way that a preacher could quote a passage but not explain it or amplify it so that its meaning becomes clear. Could it be then, that a good hymn or song is rather like a good sermon, where a text or doctrine is set out and amplified?

Well, almost! Clearly the medium is not speech, so we are not producing texts for sermons or poetry. The medium is music and so this dictates the kind of text we must use. It has to be sung, either to one another or to the Lord. Nevertheless, there needs to be a development of thought that expands that truth or doctrine with singable memorable words.

One of our standards, then, is that the text must convey biblical truth in such a way that it builds up the congregation that sings it. It must mean something, and something that is intelligible. Hence we have a commitment to updating texts, whilst retaining the weight and strength of the original.

In reporting on the work of Jewish composers (R M Campbell, Jewish music exploration widened composer's palette, September 2002) notes a statement of Daniel Asia:

Setting a text to music demands a kind of reading of the text that is much deeper than when you quickly read them. I am with the words for a bunch of days, figuring out what their speed is and their relationship to one another.

Fidelity in pattern recognition is a concern central to the challenge of sonification of scientific data as explored by the International Community for Auditory Display (ICAD) which was notably responsible for the Sonification Report: Status of the Field and Research Agenda (1997) on behalf of the US National Science Foundation. One current approach is that of iSIC using information music as an alternative to remotely monitoring complex systems like communications networks -- by exploring the use of musical rules as a way to convey information.

It is appropriate to note that a number of experiments have been made in representing molecular structures as music. In one case, as noted by M. A. Clark (Transcriptions: the music of protein sequences, 2001; A Protein Primer: a musical introduction to protein structure; Genetic Music: an annotated source list, 2005) all of the musical sequences are "simple linear readouts of the amino acid sequences of the proteins indicated. However some sense of the protein's higher order structure emerges from the alternation between the higher-pitched polar amino acids and the lower-pitched nonpolar amino acids". A point to be made is that if the structures constitutive of the human body lend themselves to useful musical respresentation, there would appear to be a case for seeking to represent the legal instruments purportedly constitutive of the human community to music -- as has been characteristic of many faith-based communities of the past.

Distinguishing modes of aesthetic sonification

The possibilities to be discussed can usefully be distinguished in terms of their position within the following table. The unexplored zones of value are Zone A preceded by Zone B.

4 Zones of exploration and associated dimensions
potentially
"interesting"
mnemonic gestalt/integration
content fidelity
interesting synthesis
cognitive
entrainment
compression [B]
semantic signature tune
("tonal barcodes")
content image maps
.
[A]
mnemonic/semantic
enhancement
gestalt comprehension
.
elaboration/extensive
inspiration fidelity
agreeable representation

[D]
encoding
semantic analysis/mapping
.
.

[C]
"setting to" poetry/music
accompaniment
inspirational celebration
EU: Ode to Joy

technical
representation
linear/sequential
unmemorable as a whole
aesthetic
appreciation

Notes:

Conventional aesthetic enhancement and adaptation of constitutive documents

Under ideal circumstances, it is indeed clear that legal texts may be set to music (possibly with sung lyrics), adapted into poetic form, portrayed as an image or even dramatized -- Zone C in the table above. This exploration avoids consideration of these entirely feasible possibilities for a range of reasons -- previously articulated to some degree (Aesthetics and Informatics: the art of information for policy-making and community-building, 1999).

The challenge of this conventional mode may be described in terms of the following:

Especially interesting as a constraint is the dynamic debate within the arts between a work as an individual creation (including when a group is orchestrated by a director) and those contexts in which collective improvisation is cultivated -- as in some jazz and dance groups. An international legal text is typically the result of collective endeavour through which challenging collective dynamics are no longer the primary focus as they may well remain in the relationships between creative artists.

Interrelated modules, challenges and applications

Notably as demonstrated by the commerically successful approach taken to the various Kurzweil applications, there is a range of partially interdependent modules (many already developed) through which the challenges can be addressed and which may give rise to innovative products.

A distinction can be usefully made between applications which:

The modules to be considered are:

Related initiatives

In a blog discussion thread on the theme of a Chess to music translator (Halfbakery, 2004):

This then is speculation that a method of converting chess notation to music would produce music that was recognisable as a particular game and that when playing a game it might be possible to hear the game in your head and possibly hear a tune similar to a tune from another game.

Partners in Distance Learning (Patriotism Expressed Through Song) offers courses in which:

The historical and cultural background in connection with writing patriotic songs will be discussed, demonstrated and critiqued. Students will be informed of the principles of setting text to music. How to compose music using centonization and tendency-tone resolution will be explained.

Eva J. Egolf (Digital Storytelling With Music, 2007; Setting Text to Music) advocates to students the use of GarageBand or Audacity software to set text or poetry to music. The text can either be student generated or selected from a work studied in class. Text generated using Song Form (with alternating verses and chorus), Narratives, and Poetry work well. When setting text some musical decisions will have to be made. Egolf notes the following considerations:

1. Mood: The mood of the text will influence which type of musical sounds accompany it.
2. Tempo: The speed of the music is also influenced by the mood. Setting a tempo at the beginning of the song is important in the recording process.
3. Form of the Text: The form of the text will influence the form of the music. The text usually should be finished before trying to write the music to it. The approach to creating music will depend upon whether the text is in song form, narrative, etc.

Roger Alsop (Enhancing the Emotional Impact of a Text through Electronic Manipulation) comments on experimental processes taken in creating a musical composition (using the IRCAM Signal Processing Workstation) to be performed in conjunction with a poetry reading:

The result is a text setting in which the composition and the accompanying reading share equal importance in performance....The entire sound track was made up of segments of readings of the text and other vocal utterances by Felix Nobis. These segments were adjusted in the ISPW Max program, using pitch shifting, ring modulating, sampling and delaying algorithms. The resulting modified voice sounds were assembled using the digital audio editing program ProToolsª to create the final piece..

The goal of this piece is to produce a composition in which the emotional aspects of the text used is reinforced and commented on through computer-based electro-acoustic manipulation.

Words, when uttered, contain certain meanings not related to their lexical meanings. Physical, emotional and mental states affect the manner of utterances and therefore their perceived meaning. These effects are usually in the pitch, timing and spectrum of the utterance, that is, the musical aspects of the utterance. For complex communication differences in pitch, timing and spectrum alone are not as useful as utterances containing meanings agreed upon by the communicants. However it is often the pitch, timing and spectrum of an utterance that is most effective and most noticed when speech is the avenue of communication.

Proposals (probably speculative):

Ebay-Generator (Ubermorgen.com): Proposes (, an contemporary European techno-fine-art avant-garde) conversion of Ebay user profiles into unique songs:

"We will create individual user profiles - based on a large quantity of data from Ebay - and transform these into unique songs. We scan Ebay sellers and buyers for their rating, sold objects, times and frequency of transactions, product sources and further data and automatically (by using custom made data-retrieving and -mining software) transform this data into a structured text. Based on this text we generate Text-to-Music with a special supercollider-application and the lyrics on top with the same text by speech synthesis. The songs will be part of the Ebay-Generator Web-Site (each user can generate text/songs by entering an Ebay-username)". [more]

A Slate or Myslate application is indicated by Michael Kinsley (Introducing MySlate: It slices, it dices …, 2000) as having been proposed within Microsoft "allowing users to listen to musical interpretations of their favorite Slate features (using Microsoft's patented text-to-music software)"

The US Declaration of Independence has long been celebrated with music. Shakespeare and Company (Lenox MA), for their 2007 season, propose a rendering of the Declaration of Independencer in the following terms:

Hear the powerful rhetoric that gave birth to democracy in America, delivered with eloquence that Shakespeare would have loved. Inspired by the philosophy and intellectual ferment of Renaissance England, Thomas Jefferson penned a tract as far-reaching and controversial as history itself.

Conversion options (steps)

1. Prose to poetry: Some steps in this process should benefit from past experiments in computer-generated poetry. Examples include:

For example, a key aspect of the Gnoetry software is the ability of a human operator to intervene in the language generation cycle, helping to "guide" the artistic process and to produce a result that is a true collaboration of equals. Gnoetry's user interface allows the human co-author to regenerate text on all linguistic levels (from word to entire gnoem) to make changes to the program's (blank verse!) analysis of the language corpus, the language corpus itself being a statistical derivation of one or more texts. Analysis renders the text(s) into three-word language tokens which are then awarded or penalized based on their ability to fit into a chosen poetic form (from haiku to sonnet). The results below were not edited post-production; all the language here is machine-made, although there is a not-insubstantial syntactic inheritance from the source texts that has survived the stochastic processes. (Eric Elshtain and Jon Trowbridge. Gnoetry 0.2 and the Transcendence of the Human Poetic, January 2007)

The Kurzweil Cybernetic Poet (KCP) attempts to mimic human creativity by using English in a very well defined stylistic manner. As described by Kurzweil

KCP is a computer program and it's provided with an input file of poems written by a human author. The program analyzes these poems and creates a word-sequence model based on the poems it has just read. It then writes original stanzas of poetry using the model it has created. (Kurzweil 374)

2. Poetry text to Voice/Speech: Software developed to provide spoken output from text can (optionally) be used for audio output of the result of the previous conversion. Attention needs to be given to the enunication of the text as poetry in contrast with the proven use of such applications for prose. Options include the possibility of:

3. Poetry to Song (or Chant): Several possibilities should be open to consideration by users

a. Minimal backing: This option would enable the poem to be spoken with a musical accompaniment, with some points of connectivity between the structure of the music and that of the poem. The option should enable a choice to be made between styles of music and the desired aesthetic effect.

b. Chant form: This option would focus on reducing the verses to a form that would emphasize a highly rythmic chant. One variant is that used by the Finnish male Shouters Choir. Others include battle and work chants, as well as rap

c. Simple song: This option would emphasize the configuration of the lines and verses to suit a selected style of song:

d. Song with chorus: This option would allow an overriding theme to be repeated in chorus form between verses. Users could be offered the option of specifying thematic keywords appropriate to such a chorus

e. Polyphony: This option would involve one of several variants, notably as a means of handling the perspective of different stakeholders bound together by the text:

4. Poetry to music (only): In this variant lyrics would be dropped and replaced by a melodic form in which leitmotivs were appropriately selected to represent the distinct issues

Design considerations

1. Degree of interactivity with user: This caqn be kept to the minimimum with specification of options (as noted above) or extended to allow the user to approve choices (eg in rhyming or synonyms), and amend or tweak provisional results in a sequence of iterations

2. Resources and insights from poetic and musical traditions: Clearly there is the possibility of orienting the text in the light of musical traditions and styles: Fadas, Reshas

3. In some cases there may be a strong case for embedding slogans in the texts. These could be extracted from a thematic database ("Peace Now", "Health for All by the Year 2000", "Spreading democracy", etc) including lines of fact ("100,000 children dead"). The slogans could (optionally) be offered as a counterpoint to optimistic promises in the main lyrics, where such promises have been broken in the past.

4. The capacity to provide a political orientation to the presentation of the verses or lyrics, or to the style of music, could be of value.

5. Selection of song or music as a template: Given the techniques of morphing images, there is a case for exploring the possibility of allowing the user to specify a song or piece of music as a "template" onto which to project the converted text.

6. Thematic leitmotivs: Under some circumstances there would clearly be a case for allowing the user to specify or select from options a melodic leitmotiv to be associated with a particular phrase (sustainable development", "freedom of information", "human rights", etc) so that their interplay can be highlighted in musical renderings.

7. Systemic issues (feedback loops): As a technical document it is appriopriate to assume that a treaty or a declartation is effectively a pattern of checks and balances whose relationship could be traced out schematically (using an appropriate methodology). The merit of the aesthetic relationship between the elements of the converted rendering is that such systemic links can be held by aesthetic devices as mnemonic aids to comprehension of the whole. An excellent example is provided by the Biochemists Songbook whose songs chart out the operation of complex metabolic pathways using popular tunes as a characteristic template for each "cycle".

8. Personalization of international treaties: The assumption is made that a legal text is a rigid text. It is however possible to understand the pattern of links articulated by that text as the focus of the agreement rather than the sequence of words. This is of course evident when the text is translated to another language. In this case the focus is on translation to another medium. There is no reason why aesthetic adapations, possibly adapted for different cultures and aesthetic preferences, should not carry the same invariance through other means. This even suggests the possibility of allowing end-users to personalize such adaptations of tr4eaties whichj apply to them so that they are meaningful in their own aesthetic language.

9. Media implications: The possibility that a "dry" treaty should be expressible in poetic form, song or music might focus communication with the media on the "signature tune" of some such treaty. The possibility that it might be "sung" by the signatories would of course be an exaggerated expectation -- but it might be sung by the analogue to a "spokesperson", a "policy singer" or poet as has been traditional in a number of cultures.

10. Singability and danceability: The above options suggest the possibility of producing singing and dancing variants of the EU constitutional treaty. These would be a most significant step in rendering any such document meaningful.

11. Hypertext poetry and related visual possibilities: Further developments include the use of hypertext to position the elements of the poem in two or three dimensions. It offers the possibiloity of other forms of order and the associated mnemonic possibilities. The elements of the displayed poem may also move. The poem can either involve set words, phrases, lines, etc. that are presented in variable order but sit on the page much as traditional poetry does, or it can contain parts of the poem that move and / or mutate.

12. Animated renderings of institutional programme agreements: Elsewhere a proposal was made to make use of the interactive multi-user SodaConstructor facility that enables animated structures to be built over the web (Animating the Representation of Europe Visualizing the coherence of international institutions using dynamic animal-like structures, 2004). It was argued that such structures could be generated from a database of the line items of the budget of an international programme. SodaConstructor also offers the facility of associating sounds with the movement of such structures.

13. Sonification of web content and e-mails: This facility has been explored. The issue is the extent to which the sonification highlights new topics in a manner that is relevant to the user and whether this could be adapted to other texts. There are clearly resemblances to the detection of thematic content in emails for the purposes of displaying meaningful adverts (as with Google).

14. Statistical improbable phrases: Amazon.com notes for user consultation the "Statistically Improbable Phrases", or "SIPs", that arose as the result of scanning the content of books offered for sale. These are the most distinctive phrases in the text of books in Amazon's Search Inside!™ program. If this finds a phrase that occurs a large number of times in a particular book relative to all Search Inside! books, that phrase is a SIP in that book. SIPs are not necessarily improbable within a particular book, but they are improbable relative to all books in Search Inside!. Clearly this could be the basis for sonification cues, perhaps personalized by the user.

15. Sonification of hit lists and reference book pages: There is a case for exploring the sonification of search engine hit lists so that mouseover triggers comprhensible sonic feedback providing a semantic soundscape of the contents. A similar approach could be taken with reference book pages (eg Wikipedia), effectively giving each page a useful signature tune.

Possible texts for experiment

There are many focal texts that lend themselves to experiment with combinations of the above modules and processes. These might notably include:


References

Roger Alsop. Interrelating the Use of Computers in Music and Language. Thesis, 1992

Denis L. Baggi:

Robert S. Boas. DNA and Protein Music. Center for Genomics and Human Genetics, 2002 [resources]

Jordan Boyd-Graber. Semantic Poetry Creation Using Lexicographic and Natural Language Texts, 2000 [text]

Jim Carpenter. LTAG, you're it! 26 December 2006 [text]

Chris Chafe. A Short History of Digital Sound Synthesis by Composers in the U.S.A. [text]

Elizabeth M. Colechio and Sara K. Hartley. The Effects of Lyrics on Melodic Memory. 2003 [text]

Joseph Coroniti. A Relationship of Equals? Aesthetic Theories on Setting Poetry to Music. In: Poetry as Text in 20th Century Vocal Music. Lewiston, NY: Edwin Mellen Press, 1992

Eva J. Egolf:

Gerald England. Poetry and Computers. Fractal Report, 6, 1994, 31 [text]

Karlheinz Essl. Computer-aided Composition. 1991 [text]

Ivan Fonagy and Klara Magdics. Emotional Patterns in Intonation and Music. In: Dight Bolinger, ed. Intonation. Ringwood Victoria, Penguin, 1972, pp. 286-312.

Charles O. Hartman. Virtual Muse: experiments in computer poetry. University of Press of New England [for] Weleyan University Press, 1996. [contents]

Tom Hinkle. Symmys. [text; computer-generated poetry]

David Humphrey. Toward a Definition of the Code Poet [text]

Heiko Idensen. Poetry should be made by all: from hypertext utopias to cooperative net-projects netculture - cultural networks [text]

Anthony Judge:

Lazarus Corporation. Text manipulation, cut-up technique and computer generated writing links [text]

Jeffrey Little. The Babble Poems. Mudlark No. 22 (2003) (poems written in collaboration with the BABBLE text generator)

Nandy Millan. References on Computer Generated Poetry and Visual Arts [text]

Sergei Petoukhov. Music and genetic code, 2003 [text]

David Rosenboom. Propositional music: on emergent properties in morphogenesis and the evolution of music, 1997

Marlys M. Styne. Poems by Computer: introducing poetry in a high-tech society, 1986 [abstract]

Johan Sundberg. 'Speech, Song, and Emotions. In: Manfred Clynes, ed., Music, Mind and Brain. Plenum Press, 1982

United States Patent:

Marius Watz. Resources Related to Computer-Generated Writing. 1997 [text]

Chong Yu. Computer Generated Music Composition. MIT, Thesis, 1996 [text]

Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.

For further updates on this site, subscribe here