LIGHT.VOID~ : AN AUTOGRAPHIC CASE STUDY ON APPROACHES, CONSTRAINTS, AND AFFORDANCES WITH A LIGHT-DEPENDENT MUSICAL INTERFACE

: This paper discusses the strategies, considerations, and implications of designing and performing with a light-dependent digital musical interface (DMI), named light.void~ . This interface is introduced as a replica of light thing , an existing DMI designed and popularized by British artist Leafcutter John. The rationale for reproducing this DMI is presented, followed by a discussion around the guiding criteria for establishing data-to-sound mappings, and the kind of affordances that these decisions may bring — including performer control, unpredictability, intentionality, spontaneity, action-sound reactivity, visual interest, and so on. The remainder of the paper focuses on dissecting the nature of this digital musical instrument, using contributions by DMI researchers Miranda and Wanderley as the main analytical framework. The outcome of this process is a semi-improvisational work titled «Umbra», along with the open source documentation for the light. void~ interface. Additionally, some relevant questions emerge with regards to performer expertise, observed vs. unobserved performance, as well as ontological frictions between instrument, composer, performer, designer, and audience.


Introduction
Mainstream scholarship on improvisation, aleatory, and experimental music is often preoccupied with at least one the following subject matters: the composer, the work, the performer, or the listener. Significantly less attention, however, has been paid to the role or influence that the instrument itself may or may not have in these contexts. Such omission is an understandable and reasonable one insofar as one is dealing exclusively with traditional, acoustic, Western, and even Eastern instruments, with a long and well-known history of performance practice. However, our ontological understanding of what an instrument is has been gradually redefined and expanded by recent developments of art-oriented technologies. Creative programming environments, such as Arduino, MaxMSP, PureData, SuperCollider, ChucK -to name a few -have played a decisive role in allowing artists and programmers alike to establish, on their own terms, the sonic potential of a wide variety of physical objects -even those that we would not often assign musical meaning to, one of many examples being Perry Cook's JavaMug (Cook, 2017, p. 5;Wang, 2018).
An increasingly large portion of this practice has been impelled by what is now known as digital musical instruments or DMIs (Miranda & Wanderley, 2006): sensor-dependent objects that allow for user-defined mappings of captured data -e.g., motion, pressure, lighting, temperature, etc. -onto different kinds of digital sound processes, affording unconventional ways of real-time, human-computer interaction.
This paper focuses on a specific instance of DMIs, here referred to as light.void~: a black acrylic enclosure with a 4x4 matrix of photo-resistors, each of which sends lightsensitive data to the computer via serial communication. First, a general overview of light-dependent DMIs is provided. Then, the light.void~ interface is introduced as an intentional replica of light thing, an existing DMI designed and popularized by British artist Leafcutter John. The rationale for reproducing this DMI is presented, followed by a description of the main strategies taken for gesture acquisition in light.void~, the decision-making process at the time of establishing data-to-sound mappings, and the kind of affordances that these may bring -including performer control, unpredictability, intentionality, spontaneity, action-sound reactivity, visual interest, and so on. The discussion around these issues is informed by recent contributions in DMI research, using of the work of Miranda and Wanderley (2006) as the main analytical lens. The outcome of this process is a semi-improvisational work titled «Umbra»¹, as well as the open source documentation of the light.void~ DMI design².
Additionally, some relevant questions arise in relation to performer expertise, observed vs. unobserved performance, as well as ontological frictions between instrument, composer, performer, designer, and audience.

Light-dependent DMIs
The idea of visible light³ as an interactive medium has appealed to many artists since the early developments of electricity during the 19th century. One example is the Chromola or color organ, used by A. Scriabin in his 1915 tone poem Prometheus: Le Poème du Feu (van Campen, 1999, p. 10). This interest is especially evident in the context of DMI research, where visible light has been widely used both as a causal and/or reactive mechanism for human-computer interaction. Broadly speaking, light-dependent DMIs can be described as having two main interaction modalities: projection and interruption of light.
One instance of these light-dependent DMIs is the LightHarp: a MIDI controller that "uses spotlights and lasers to trace virtual strings through space" (Favilla & Cannon, 2006, p. 372). This instrument includes a series of supplementary controllers, offering up to 5 degrees of freedom available to the performer, where the main mode of interaction is the interruption of light beams (i.e., the virtual strings) as a means to trigger events. Another instance is the Light pipes, which are "a series of pipes that respond to incident light" (Won et al., 2004, p. 209), generating MIDI messages that are later mapped onto different pitch-based algorithms in PureData⁴. In this case, the performer is meant to control the interface by projecting light onto the pipes, where the sensors are located.
There are also instances in which these two interaction modalities are used in tandem within the same DMI, such as in the Light instrument (Eyes & Jongejan, 2016), which implements a feedback system whereby sound and light act as control mechanisms of each other, using field recordings as the main sonic component. More precisely, the sound's amplitude acts as control for the intensity of an LED, which in turn controls the amplitude of the sound. Therefore, both interaction modalities are possible, by either projecting or interrupting light to respectively increase or decrease the amplitude of the field recordings.
Another interesting example is The Pearl (Hattwick & Wanderley, 2015), a spherical light-reactive instrument designed as a prop for a theatrical performance. Different from the previous examples, light is not used here as a control mechanism, but rather as a way "to display information regarding the state of the performance system, to convey information regarding performance gesture, or to add aesthetic elements to the performance" (p. 203).
Lastly, there are two DMIs that, despite sharing the same name and having a very similar design, remain different in crucial ways: Pak's Light Matrix (2006) and Fieldsteel's LightMatrix (2018). Pak's Light Matrix follows a similar approach to the Light Instrument, in that it uses a feedback system of a 8x8 matrix of LEDs which acts both as "a proximity sensing array and a monochrome display" (Pak, 2006, p. 342).
³Uses of light wavelengths outside the visible spectrum, such as infrared light, are outside the scope of this paper. ⁴ https://puredata.info/ Thus, the instrument projects light onto itself, by using the performers hands as a reflective surface, and the amount of light detected is used as control data for DSP mappings. One of the peculiarities of this DMI is the fact that the 8x8 arrangement of LEDs is not used as a two-dimensional interface, but rather as a horizontal array, where each column controls the gain of a synthesizer's frequency bands, arranged from left (low) to right (high) -in other words, the implementation is purposefully one-dimensional. On the other hand, Fieldsteel's LightThing consists of a grid of 16x16 photoresistors where the main mode of interaction, at least as shown in Fieldsteel's work, Depth of Field (2018), is by setting up a desk lamp above the matrix and manually interrupting the light from reaching specific regions. The absence of detected light is then used as control data for several DSP parameters, such as amplitude and filter bandwidth (Fieldsteel, 2022). These two interfaces are perhaps the most similar in design to light thing and, consequently, to light.void~. The decision to reproduce a pre-existing DMI, especially one of which not enough information is known, can be explained by considering preliminary questions that the endeavor itself raises. In DMI research studies⁷, focus groups are often used to both evaluate and analyze user experience, by testing the same interface with different participants. Thus, the subject acting as a user or performer is variable while the interface and, crucially, its designer or manufacturer, remain constant.

The interface light.void~ and light thing
This allows researchers to observe and detect more easily any emergent behaviors and patterns in the interaction between participants and a given DMI. However, in the case of more traditional electric instruments, it is worth considering the impact that design variants or models may have on the types of emergent interactions between instrument and performer. For instance, with the electric guitar, cumulative changes in design by different manufacturers -e.g., fret size, wood type, body shape, weight, pickup arrangement, and so on -can have a noticeable influence over its affordances, without compromising the ontological integrity of different guitar models as being the same instrument. This brings up the issue of replicability in DMI research, recently discussed by Calegario et al. (2021), arguing that "improving the documentation and replicability of our musical artifacts should help us avoid "reinventing the wheel" and instead focus our efforts on either a) under-explored regions of this design space or b) tweaking designs that have been deemed interesting or 'successful'" (pp. 2-3). For this reason, this paper seeks to both explore aspects of design and performance in light thing, through its inferred replica, for which the open-source documentation is included in the form of a Github⁸ repository. This will facilitate the replicability of this DMI by other researchers, designers, and artists, and further elucidate its full range of capabilities. After all, "replicability strongly impacts the difference between a "living" instrument and an archived one" (Calegario et al., 2021, p. 4).

Basic functionalities and methods for gesture acquisition.
The uncertainties regarding the design of light thing were taken as an opportunity for envisioning and experimenting with a variety of mapping approaches, informed by direct and indirect experience with the light.void~ interface. In order to understand the chosen approaches for gesture acquisition and data mapping, it's important to mention an additional, creative motivation behind building this interface, namely that of approaching body movement and physical gestures as the paramount criteria during the compositional, improvisational, and performative process. Similar to Leafcutter John, the preferred medium of interaction was the use of different kinds of small, portable, handheld flashlights, which visually contributes to highlighting arm movements in dark performance venues. With this goal in mind, a few but quite effective ways of gesture acquisition emerged.
The mechanics of the light.void~ electronic circuit, its Arduino code, and the MaxMSP user interface are fairly simple: each sensor sends 10-bit data values to MaxMSP, where these are normalized to floating-point numbers between 0 and 1, after being independently calibrated according to perceived levels of ambient light in order to avoid base-level noise. The greater the intensity of the light perceived by each sensor, the greater the output value is. This results in 16 independent data streams available for user-defined mapping.
As previously stated, one of the goals of this project was to take the choreography involved in sound production as a point of departure to decide what the sounds themselves would be. This led to the use of video and motion data recordings as the primary strategy for experimenting with different data-to-sound mappings and gesture detection (see Figure 2). Each video contained a series of movements or gestures that were of particular interest for their visual appeal and gave a general idea of what the desired mode of interaction with the interface would be. Crucially, these movements were enacted without any specific sound mapping in mind. From this process, three algorithmic abstractions were written in MaxMSP to work as a reusable toolkit for establishing mappings between gestures and DSP algorithms.
Each abstraction had a specific set of functionalities with gestural correlates, and are listed here using the names given to each of them: lv.data, lv.timetrigger, and lv.centroid.

lv.data
The lv.data abstraction allows for user-defined clusters or groupings of sensors within the matrix to be treated as a single, unified control region. The resulting region is then represented as a single value or parameter variable, defined as the current maximum value between the sensors belonging to that region. The most recurrent implementation of this abstraction was for creating subdivisions of the matrix in the form of quadrant regions -e.g., subdivisions of the sixteen sensors into four 2x2 matrices. This brought more flexibility and reliability to the performance, since sound control parameters that depended on single sensors proved to be quite unpredictable, particularly with spatially wide gestures. For example, a region consisting of all 16 sensors was be a reliable way to capture the proximity of the flashlight -and by extension, the hand -to the interface. In «Umbra», this was consistently used as amplitude control for different DSP algorithms.
lv.data also calculates the velocity -i.e., the first derivative -of the control region value. The velocity is sent through two different output channels depending on whether it is a positive -i.e., increase in light signal -or negative value -i.e., decrease in light signal. When either the negative or positive velocities cross a given threshold, which can be defined by the user, the abstraction outputs a message intended for event triggering. This was useful for detecting intentionally abrupt or fast hand/arm movements, while also allowing to differentiate between inward and outward motion.
The two primary uses of this feature in «Umbra» were triggering one-shot samples, and randomizing or updating DSP parameters.

lv.timetrigger
lv.timetrigger works as a hold-and-release switch, intended for single-sensor use, as opposed to the previously outlined region-oriented approach. It consists of a doublethreshold system, one threshold being light-sensitive and the other time-sensitive. The abstraction is assigned a sensor, and it waits for the light level to go above the lightsensitive threshold -when this threshold is crossed, it starts an internal timer. The timer continues running for as long as the light level remains above its threshold. If the elapsed time crosses an established time threshold -say, more than 500 milliseconds -, a trigger message is output once the light level goes below the threshold again.
If the light level goes below the light-sensitive threshold before the time-sensitive RICERCARE Revista del Departamento de Música -Grupo de investigación en Estudios musicales Núm. 15 (2022) threshold is crossed, no message is generated. This is the light.void~ equivalent of a button, which in «Umbra» was used primarily for changing sound-mappings, and consequently contributing to defining the form of the piece. For a 3-dimensional position tracking, the 16-sensor region approach outlined above can be used in combination with lv.centroid to provide a z coordinate -i.e., height, or vertical proximity to the interface. Although this abstraction was developed after «Umbra» was already considered a finished work, it has also been tested for gesture recognition using machine learning algorithms in MaxMSP (Françoise et al., 2014), with relative success. Another potentially effective implementation of this abstraction is real-time sound source spatialization, where the position of the flashlight with respect to the interface controls the location of the sound source relative to the speakers.

Digital Signal Processing
While an in-depth explanation of the DSP algorithms used in «Umbra» is far beyond the scope of this paper, these were a combination of corpus-based concatenative synthesis (Schwarz, 2011), trigger-based sampling, wave-folding synthesis, resonant filter models, a simplified version of Tom Mudd's gutter synthesis technique (Mudd, 2017), among many others. The different real-time control strategies for these DSP algorithms were a combination of continuous and discrete control modes. Examples of continuous control were the use of sensor region values to control loudness or textural density -e.g., the greater the proximity of the flashlight to the surface, the louder the sound or the more active the texture becomes. Examples of discrete control included event triggering through the kind of velocity transient detection outlined earliere.g., abrupt hand gestures would either trigger a sample or randomize/reset specific parameters -, and the hold-and-release functionality used for changing from one predefined mapping to another.
Felipe Tovar-Henao light.void~: An Autographic Case Study on Approaches, Constraints, and Affordances with a Light-dependent Musical Interface.

Constraints and affordances
In their book New Digital Musical Instruments: Control and Interaction beyond the Keyboard, Miranda and Wanderley (2006) offer a variety of useful concepts with which one can describe, compare, or analyze distinct features of a given DMI. Along with these concepts, a basic DMI model is introduced which consists of a network of two units -a gestural controller and a sound production unit -which communicate through gestural inputs, two types of feedback sources, and several mapping strategies (see Figure 3). Comparing it to this model, one of the glaring differences with light.void~ is the absence of the primary feedback channel, which in most cases would be tactile or some other kind of force or resistance feedback -e.g., the vibration of a string, the air pressure of a wind instrument, pressing down a key, etc. While there is physical interaction involved through the manipulation of flashlights, this physicality does not inherently provide any meaningful information to the performer about the current behavior or state of the instrument, let alone anticipate the quality of the sound. The interaction is then mainly dependent on hand or arm gestures and, if any primary feedback source were to be argued for, it would have to be a combination of visual cues and muscle memory -the latter assuming that the implemented strategies for gesture acquisition are satisfying enough to establish a strong visual link between different types and gradations of muscle tension and their corresponding mappings. In addition to the primary and secondary feedbacks, two other classifications are considered, namely passive and active feedback -the former being "provided through physical characteristics of the system (e.g., the noise of a switch)", and the latter being "produced by the system in response to a certain user action" (Miranda & Wanderley, 2006, p. 11). In relation to these added features, the passive feedback is also missing in the light.void~ design, given that no sound is produced when interacting with the system, unless we were to consider the flashlights -in particular, the sound they may produce when turning them on and off -as part of the instrument itself.
Another current shortcoming of the design compared to other DMIs and acoustic instruments in general, lies in the restrictions it poses when interacting with two different hands, or more specifically, with two different flashlights. While the sensors certainly respond to light intensity regardless of the number of sources, the only way of capturing differences in gestural behavior between the two hands is (a) if each of them is restricted to controlling separate sensor regions, and (b) if the mappings are specifically programmed to capture independent gestural information in those separate regions. This influenced the general mode of performance interaction in «Umbra», assigning most movements and gestural control to the right hand, with a sparing involvement of the left hand for mapping changes with a separate flashlight and other complementary functions such as covering the right hand's flashlight to 'mute' the sensors more accurately.
Although one possible way of overcoming this limitation in 'hand labor distribution' might be through the incorporation of wavelength -i.e., color -sensitivity in the design, the goal of acquiring an ideal 'gestural polyphony' might still present some technical challenges. Additionally, the gestural frequency range -differentiating between consecutive fast gestures -proved to be relatively limited⁹ especially when using some of the intermittent settings of the flashlights. It is worth clarifying, however, that the purpose behind making these observations and criticisms is not that of undermining the interest or potential of this DMI, but rather to establish some boundaries which might help delineate or point to a theoretical or conceptual framework of performance practice, expertise, and technique acquisition specific to this interface.

Performance
In discussing gesture, Gadd and Fels (2002) introduce a framework of mapping transparency, which can serve "as a predictor of the expressivity of musical devices", and where "expressive instruments are those that effectively convey the feeling of a player to an audience" (p. 1). In other words, transparent mappings are those in which the gap between the performer's intent and the fulfillment of that intent is sufficiently reduced. One alternative favored by the authors is the use of design metaphors, whereby "cultural bases or elements that are "common knowledge"" (pp. 1-2) can aid the audience in understanding the correlation between performer's gesture and the sonic result. An example of this is the MetaMuse DMI (Gadd & Fels, 2002) in which tilting a watering can over a prop surface works as a metaphor to control higherlevel features of granular, water-like sounds. Here, the causal link between gesture ⁹ Around a 10 Hz detection threshold. and sound is immediately clear, as it relies on common or familiar knowledge on the audience's part. Although the grid-based design of light.void~ does not offer meaningful opportunities for these kinds of metaphors, the framework of transparency was of particular importance leading up to the performance of «Umbra».
As was mentioned before, the main strategy during the trial-and-error stage with gesture-to-sound mappings was the use of video and motion data recordings to test their efficacy. Similarly, the rehearsal process for the first performance of the work involved audio in addition to video recordings as a means to evaluate the overall quality of the sound, the variety in gestural vocabulary, and the correspondence between these two. While this contributed to making the process much more efficient, it also meant spending less time using direct interaction with the interface as a criterion for evaluating gesture-sound correlations -in other words, the decision-making process was mostly guided by disembodied, as opposed to embodied, listening. An unsuspected result of this was that some of the final mappings proved to be much less tolerant or susceptible to performance error, thus reducing the impact of some of the videorecorded gestures. While this certainly guaranteed a higher level of consistency and robustness during live performance, a fair amount of gestural information present in the videos became rather unnecessary and could have been removed without substantially affecting the sound reactivity of the DSP algorithms.
Nonetheless, the self-perception during performance when these gestural components were absent or replaced, even if the sonic result was for all intents and purposes the same, felt qualitatively different: it lacked transparency. The same could be said about the use of flashlights with different wavelengths, which had little to no effect on the overall sonic result but otherwise contributed to visually highlighting formal boundaries. This posed a critical question on whether these gestures were indeed nonessential or, on the contrary, brought a kind of theatricality that was integral to the effectiveness of the performance or, at least, of this specific work.

Conclusions
In this paper, a light-dependent DMI was introduced, along with its documentation and a semi-improvised work titled «Umbra». The interface was presented as a replica of Leafcutter John's light thing, of which not enough information is known, in the hopes of facilitating more opportunities for DMI replicability. A detailed discussion about its features, affordances and shortcomings was included, raising worth-while considerations about DMI design and performance practice.
The dependency on light as the primary medium for DMI interaction poses unique challenges to the performer and, in some ways, requires a cognitive readjustment with respect to what can be considered technically idiomatic. At the same time, the established mappings for «Umbra» allowed for a relatively low 'entry fee', which is one of the features generally sought after in DMI design as a means of democratizing performer expertise (Jack et al., 2018). For instance, similar to a piano or keyboard instrument, it is difficult to produce a sound drastically different from what it is supposed to produce, especially when the established mappings have a lower tolerance in sound variability. In return, this may afford a space for a faster learning curve while enabling careful thought and deliberation in determining what the gestural vocabulary in performance will be, so as to provide the desired aesthetic experience.
Possible avenues for improvements in gesture mapping include implementing the cross-coupled parametric approaches described in Hunt et al. (2002), as well as using the taxonomy introduced in Levitin et al. (2002) to improve and enrich control over musical events. Furthermore, implementing a focus group study in the future might be beneficial in observing and detecting emergent behaviors in user interaction with light.void~, by experimenting with different mappings and models.
Given that the sensors only react to light intensity, it is possible to produce similar data streams with entirely different actions. For example, gradually reducing the light intensity with a flashlight dimmer or slowly decreasing the proximity of a flashlight with invariable intensity from the surface, can both virtually produce the same data, and by extension, the same sound. Yet the qualitative difference between these two gestures and the way they may be perceived is nontrivial. Whether piece-specific or DMI-specific, it is worth considering ancillary gestures as potential semiotic devices in performance practice.
Different degrees of unpredictability may arise depending on factors such as performance space, design, and behavior of chosen flashlights -e.g., beam focus, size, number of settings, etc. -, post-calibration range, and likelihood of changes in ambient lighting during performance. These circumstances, however, can also be thought of as boundaries within which technique acquisition can take place, which may emphasize fine gestural control and embodied awareness. However, it is unlikely that the 'ceiling' of dexterity could be as high as with, say, a piano, not necessarily from the performer's end but from the technological one -especially considering the aforementioned ~10 Hz limitation in gestural frequency range.
As with many DMIs, the range of repertoire one can perform or improvise around is heavily constrained by the work or works the instrument was conceived for -not coincidentally, this echoes one of Perry Cook's principles of DMI design: "Make a piece, not a instrument or a controller" (Cook, 2017, p. 3). This touches on interesting ontological questions about instrument, design, composition and improvisationespecially if we also contemplate the possibility that "in algorithmic composition, the mapping of gestures to sounds may be considered the composition itself" (Miranda & Wanderley, 2006, p. 15). Or perhaps that the strategies of gesture acquisition are what defines the instrument, which poses the question of whether Leafcutter John's light thing and light.void~ could be considered the same instrument.
Felipe Tovar-Henao light.void~: An Autographic Case Study on Approaches, Constraints, and Affordances with a Light-dependent Musical Interface.