A Blue instrument: ImagePlayer

Author: Menno Knevel

Blue has some excellent ways to generate scores by the incorporation of scripts (python, Clojure, Lisp) and Jmask, nGen, NoteProcessors and AthenaCL. More information on this subject can be found here. Using some of these scripts permit to give it a seed value, so that the user can give some direction in his/her choice as to the score pattern generated; setting the seed to a certain number assures that every time the events are generated, the pattern will sound the same way. The seed gives the user the Power Of Choice – it is not dependent on the moods of the machine...

There is yet another way to generate scores, that seems random, but in fact is not: scan an image and convert it to sound. The sounding result will depend on the image and thus is a stable entity, but the user still has ways to influence the final outcome by changing some of the parameters of the sound. You could in fact see the image as a seed: you don't know yet how an image is going to sound, but it will sound that way every time unless you change something in the image or tweak the parameters of the sound generators.

I have uploaded in BlueShare an instrument called ImagePlayer. It uses the image opcodes created by Cesare Marilungo. In the manual, have a look at http://csound.github.io/docs/manual/ImageopcodesTop.html. It took me some time to “tame” the image opcodes, so I thought I'd share my findings with the world. Of course, the way the instrument ImagePlayer is devised is – as is always the case with the Csound opcodes – just one way to use the image opcodes.

What can it do? And...how?

Good questions. I'll tell you:

• it scans an image – PNG format only
• scans the image from left to right (X- axis) and from top to bottom (Y- axis). But can be changed to move in the opposite direction.
• the higher part of the image corresponds to the lower frequencies, the bottom to the higher frequencies
• it looks for the RGB pixels in the image. That means Red, Green, Blue.
• the intensity of the red, green or blue pixel is converted to a value between 0 and 1
• the color white (all the 3 basic colors summed), will produce 1 + 1 + 1 respectively
• the sound generator uses 'adsynt2', which, according to the manual, 'performs additive synthesis with an arbitrary number of partials -not necessarily harmonic- with interpolation.'
• very low values for the base frequency (1 - 15) of adsynt2 will prohibit the (in)harmonics to be higher than the Nyquist frequency – and this causes distortion - unless you like that…So an anti-aliasing filter can be enabled and the number of harmonics can be regulated
• X axis is for scaling the frequency range
• Y axis is for limiting the screening of the Y axis of the image, so in a way controls the higher frequencies of the image
• very high frequencies may be produced. This will create aliasing and may be desirable. If not, there is a switch for turning on the anti-aliasing filter and adjust the higher part of the frequencies to your liking.

The ImagePlayer has sliders that can narrow down the width of the image so that it plays only a part of it. Or, when the start and end time is reversed, it will screen the image from right to left instead of from left to right. The XY controllers have influence on the amount of overtones and the range of frequencies. Further, the amount of each of the 3 colors of the RGB's amplitude can be regulated at k-time. But, unlike more familiar instruments that have VCO, VCF and VCA etc., this is an instrument to experiment with in order to understand what is happening.

Paint!

Instead of using existing images you can design your future sound with a third party program. GIMP is well-known and took me some time to get acqainted with. Something lighter and easier to use for drawing and painting is (on Linux!) XPaint, Pinta or Gnome Paint. I started my experiments with Xpaint; it has a cool tool called 'spray' that allows to spray clouds of colors that will later, when scanned by the ImagePlayer, become clusters of tiny sounds. The paintbrush of XPaint has features like Radius and Density and it is fun to experiment with these settings. Xpaint is an old program – some 20 years old – and it looks and feels antique, but the features of the program largely compensate for this :)

The painted PNG above can sound like this: imagesound_mono.ogg and in stereo that would be imagesound_stereo.ogg, depending on some settings and adsynt's base frequency.Perhaps not the prettiest sound you have ever heard, but it shows the sounds of different paint brushes.

If you want to start from scratch and need an empty canvas it is good to know that you must start on a BLACK canvas. Black is the color that produces only silence, so this is your “white paper”. Here is how to create a black canvas in GIMP.

• start GIMP
• File → New File
• choose image size. This will be the resolution of your “sound palette”
• Advanced Options → Color Space → RGB
• Fill With → Background Color → OK
• Right Click on the white Canvas → Tools → Paint Tools → Bucket Fill
• Choose the color black to fill the canvas anf fill the image
• Export As → Export Image As PNG
• set compression level to 0 (compression might lead to pixel distortion and that would that some unwanted sound will be heard in the ImagePlayer)

If you choose to create a “big” image, like for example 500 x 500 pixels, in principle you have created an image that can hold 500 different frequencies from the lowest to the highest frequency. An image of 32 x 32 pixels has room for only 32 frequencies.

This is a good thing: it forces you to some kind of grid, some quantization on the frequencies to be created. Remember that you can scale the frequency range later, when moving the X axis of something i called Voices (in lack of a better term). This frequency range will be build on top of the base frequency.

Here is a lovely example using a high-resolution image (500 x 500)

Listen to the result. This is a mix of all the RGB pixels in equal volume, but you can set the levels of the RGB individually – or leave out a color as you wish.

Time for a low-resolution 32 x 32 pixels example:

It is hard to see the image because of its small size, you will have to zoom in. Below, the same image, scaled 10 times, so you can actually see what is happening inside the image:

The resulting sound however is loud and clear. Notice the bigger separation of frequencies than with the high resolution image.

Back to the image; it is not easy to draw exactly one frequency. The pencil or brush must be set to 1 pixel in order to get only one frequency and even then it is likely that you will get the sounding result of a frequency range instead of only one. But this makes it an instrument that separates it from others. And yes, it is quite difficult to see it and draw a line of one pixel. You have to use the magnifier. When working with small, low resolutions images (32 x 32 for example) you won't need the magnifier because you are already zoomed in 1600% or so to be able to see what you are drawing. In this case the one-pixel-pencil is already 1/32 of the image...

I find it interesting that one can draw thick lines, so that the produced sound will be a package of neighbouring frequencies around a centre frequency. And using the jitter function is fun….

To end this chapter of the article, some words from Bob Ross: “Happy painting!”

Future plans for ImagePlayer

1. it seems feasible that, at some point, jpeg and tiff images can be imported and scanned. On most computers these libraries are already there but it takes an enthusiast developer to implement the code.
2. the image that is being scanned could be shown in the ImagePlayer instrument once all the JavaFX are implemented in Blue.
3. it might be better to calculate and fill the tables BEFORE the instrument is actually generating the sound. It would free up CPU cycles and would allow to be able to run other instruments in parallel in Blue.

Blue version 2.6.0 test5, updated december 2015