007: Towards a Theory of Invested Objects

This is the midterm: the first draft of the paper and the nut of my final written thesis. You can download as a word docdownload as a PDF, or read below.

In January 2015, Buzzfeed’s Longform section published Anne Helen Petersen’s “Big Mother Is Watching You,” a review of fitness trackers, home trackers, baby trackers, car trackers — everything trackers — and a look at the promises of quantification: that “these new devices are capturing data that used to be inaccessible and turning it into something knowable.”

Sadly, with big promises comes even bigger disappointment. While Petersen combines a look at the coming wave of devices with critical questions about what companies and governments might do with our data — and the types of stories they tell to get us to hand it over — I found myself disappointed that this is the leading edge in the development of digital objects at all. Beyond the fight to reign in surveillance or encourage it for larger goals stands the question: How did we even get here?

How did we get to Lively, the device that allows you to spy on your elder parents under the aegis of care? How did we get to the Canary Project, ready to spy on your teens whenever they are in the car (you know, for safety!)? How did we get to Point, listening to your home, Dropcam watching it, and Nest deciding exactly how warm it is? Spire to tell you you’re slouching and gloves to determine exactly who is at fault in your worker’s comp claim? Sense will watch you sleep and, at the apex the top, Pavlok, the bracelet that allows you to shock yourself into good habits.

In January 2015, my answer was simple; it was pure lack of imagination. These objects, which I like to call watcher objects, were the product of an engineering culture where our capabilities had outrun our dreams. Once we had brought Vannevar Bush’s Memex to life, the only big projects left were robotic and dystopian, and what else could we do with our sensors and networks but use them to create objects that noted and reported? And this is as much the fault of the artists as the technologists; after all, when engineers follow the traditional colonial engineering imagination (see, for instance, the “Engineer’s Imperialism” chapter of Michael Ada’s Dominance by Design), artists have to explain why we have not offered a better view of object-data-human relations or why, if we have, the engineers have not heeded it.

Sadly, this seems not to have been the case for those whose first inclination is to be critical of the technological narrative, that group Janet Murray calls the humanist. “One can think of the humanist strand as dramatizing the problem,” she writes,

amplifying our discomfort by denaturalizing the rituals by which we deny it. The disciplinary humanists in this volume, whether artists, theorists, or scholars, are all engaged in foregrounding our cultural confusions, tuning up our sense of existential befuddlement before the scientifically revealed world of the twentieth century. (4)

And yet, if we survey the current state of object-data-human relations, we see the humanist complication has had no effect in creating a better or more-human relationship. This may be rooted in the strand’s fundamental oppositionalism:

They find the punchcards of the early information age of little use. They are surveying the wreck of ideologies, coming to terms with the failed promises of print, the horrifying trajectory of the rationalist arrow. They insist that we experience the flickering focus, the slipping away of meaning between the signifier and the signified that is the intellectual predicament of the second half of the twentieth century. (Murray 4)

A refusal to acknowledge the tangible benefits technological developments and engineers have brought to the people and an inability to locate meaning in the ambiguous, however, makes it difficult if not impossible to steer the culture into better waters.

In Invested Objects then, I will investigate the types of object-data-human relationships at the foundation of our current digital culture and review some objections to this set up. Next, I will consider ways to pollute the possible with better ideas than we currently have going, including describing what better might actually mean. Finally, I will close with a look at inspirational work and other motions towards the creation of an invested object prototype. I hope that by engaging the fundamentals of a sick culture, we might find a spot to begin to unravel it.

The Problem, or How It All Began

The language and ideas we use today to discuss the relationship between machines, data, and humanity remains that of the engineers who developed digital technology in the years around the Second World War, working for the U.S. government, Bell Labs, and ARPA–funded university departments. The men — and they were pretty much all men — who laid the foundations for computer engineering worked with assumptions and aspirations based in Claude Shannon’s information theory and Norbert Wiener’s follow-on theories of cybernetics, both of which featured as topics of discussion and elaboration at the ten Macy Conferences, from 1948 through 1953.

As N. Katherine Hayles relates in her thorough and insightful investigation of the evolution of cybernetic theories, the information theory that took hold at the conferences was one that “conceptualized information as an entity distinct from the substrates carrying it” (ix) and finds it essential cybernetic expression in the notion that “humans and machines brothers under the skin.” (50)

As she explains, the arguments in the case,

were deployed along three fronts. The first was concerned with the construction of information as a theoretical entity; the second with the construction of (human) neural structures so that they were seen as flows of information; the third with the construction of artifacts that translated information flows into observable operations, thereby making the flows “real.” (50)

The idea that we see expressed in watcher objects today, that data is out there to be gathered and deployed, to be hoovered up and processed into life-changing suggestions, is predicated on this first front, the theoretical construction of information as “a mathematical quantity, weightless as sunshine, moving the rarefied realm of pure possibility, not tied down to bodies or material instantiations.” (Hayles 56)

We hear its echo in arguments that “information wants to be free,” as if information is its own form with its own desires. We see it in suggestions that properly anonymized, data is free of us, and in the notion underlying most watcher objects: that data lives in the aether to be harvested and milled.

When Claude Shannon first discussed his information theory as a mathematical theory of probability unconcerned with the content of the messages it described, he emphasized its limited applicability to problems of sending messages, as many as possible, through contemporary communications systems. (Hayles 54) However, within the context of postwar scientism, the allure of abstract and simple theory was too much to resist.

Reflexivity, the alternate theory of information, promulgated by Donald MacKay, suggests that context and structure are vital to understanding information, that “subjectivity, far from being a morass to be avoided, is precisely what enables information and meaning to be connected.” Reflexivity, however, “lost because specifying and delimiting context quickly ballooned into an unmanageable project” in the minds of researchers. (Hayles 56–57)

That is, the culture of post-war technology, metastasizing the true gains of engineering into an assertion of the preeminent value of mathematic description — abstract and simplified over messy and difficult — encouraged the adoption of a theory that refused to consider context and the production of meaning. Rather it claimed information as a free material, divorced from energy or matter. Along with these claims came an emphasis on systems as vessels for homeostasis, the closed-loop alternative to the observer-influenced and subjective reflexivity.

The conceptual wins and losses in the triumph of Shannon and Wiener over MacKay can be understood by contrasting the implications of these viewpoints. As Hayles puts it,

The price [contextless information] pays for this universality is its divorce from representation.... The price [reflexive information] pays for embodiment is difficulty of quantification and loss of universality.... Making information a thing allies it with homeostasis, for so defined, it can be transported into any medium and maintain a stable quantitative value.... Making information an action links it with reflexivity, for then its effect on the receiver must be taken into account.... (56–57)

She continues to point out that homeostatic systems reenact the same pathologization of difference as Anglo-American engineering culture.

Carolyn Marvin notes a decontextualized construction of information has important ideological implications, including an Anglo-American ethnocentrism that regards digital information as more important than more context-bound analog information. (19)

Using Norbert Wiener’s electronic rat as an example of the embodiment of homeostatic ideas and the tendency to “construct the human in terms of the machine,” Hayles writes,

Presuppositions embodied in the electronic rat include the idea that both humans and cybernetic machines are goal-seeking mechanisms that learn, through corrective feedback, to reach a stable state. Both are information processors that tend toward homeostasis when they are functioning correctly.
Given these assertions was perhaps predictable that reflexivity should be constructed as neurosis in this model. (65)

Now competing information theories are not merely a sign of different approaches, but context becomes sickness itself. As Hayles relates, cybernetics eventually moved on to incorporate questions of reflexivity and context. However, there is little evidence that industry did.

Jaron Lanier, a longtime software engineer who has contributed to early virtual reality work, picks up the argument against the conflation of machines and humanity in his manifesto You Are Not a Gadget. While establishing his engineering-insider bonafides, Lanier hits out at a contemporary culture where we accommodate ourselves to the limitations of machines and worship the fragmentary intelligence of the group.

When developers of digital technologies design a program that requires you to interact with a computer as if it were a person, they ask you to accept in some corner of your brains that might also be conceived of as a program. When they design an internet service that is edited by a vast anonymous crowd, they are suggesting that a random crowd of humans is an organism with a legitimate point of view. (4)

He ties this to a penchant for self-abdication, in this case expressed as thinking machines will know us better than ourselves, a sickness to which many prominent technologists are prone. The same dreams that Lanier critiques coming from “cybernetic totalists” like Marvin Minsky and Chris Anderson — dreams of uploadable minds and science robots who iterate on theories we no longer understand — are those Hayles identifies as the outcome of cybernetic theories. The “post-biological” hopes embraced are rooted in the initial division of information from agar, the likeness of neuron and circuit that frees data and makes it not-us.

Lanier also points at the opposition to this investment in a database accommodation of schema being the embrace of difference and context that appears at the heart of reflexivity. If fitting ourselves into databases is about accommodating ourselves to schema, we do seem to lose some of the value of being known and recognized for our idiosyncrasies. “A real friendship ought to introduce each person to unexpected weirdness in the other.” (52) While this is phrased in a more, um, Northern California manner than I might choose, the articulation is resonant.

The conception of information as a “free-floating, decontextualized, quantifiable entity” powers watcher objects and a culture that seems to shear off our idiosyncrasies. How can we fight back?

Polluting the Possible

Lanier provides one possible avenue for revolt when he describes what technologists do:

Technologists don’t use persuasion to influence you, or at least we don’t do it very well.... We make up extensions to your being, like remote eyes and ears (webcams and mobile phones) and expanded memory (the world of details you can search for online). These become the structures by which you connect to the world and other people. These structures can in turn change how you conceive of yourself and the world. We tinker with your philosophy by direct manipulation of your cognitive experience, not indirectly, through argument.” (6)

This too rhymes with Hayles’s reasoning for including the literary in her examination of cybernetic concepts in culture: “Literary texts ... actively shape what the technologies mean and what the scientific theories signify in cultural contexts.... [C]ulture circulates through science no less than science circulates through culture. The heart that keeps this circulatory system flowing is narrative.” (21, emphasis mine)

Through the creation of devices and objects that operate on reflexive principles, that value context, we can provide imaginative alternatives to the current watcher objects and thereby a possible alternative narrative of coexistence with machines — one on human terms. Through the deliberate stories we tell, we can influence the reception these objects receive. Rather that attempting to thwart a river of concept and narrative, we can add our own little drops to the torrent.

Let’s Get Specific-ish: Characteristics of An Invested Object

If we are not objects or gadgets, what kind of relationship can we have with an object and what sort would it be? Developments in digital computing have changed our world and expectations of objects — and these we should leverage. As Lanier points out above, we do now have remote eyes and ears, and that is a compelling development. To set technology and humanity at odds is a failing strategy, as we have seen the objects of “humanists” be swept away with charges of Luddism.

Lanier finishes You Are Not A Gadget extolling the “morphing” capabilities latent in VR, which when paired with human neoteny (that is, our plastic brains), will allow for us to create an augmented human future that remains human. For this he uses the humble cephalopod as metaphor and contrast.

For the invested object, I prefer a half-aquatic metaphor: the lichen. Lichen are actually two creatures, algae and fungus, combined as one. The protection of the external fungus allows the algae to live in environments too dry for survival alone. The photosynthetic properties of the algae provide the energy the fungus needs to live. It is this type of symbiotic experience I hope we can mirror with the invested object.

In this case, we can use digital technology and its veneer of the future path as the shell to protect and shepherd an experience of contextualized information. But what features can we expect from a wee investable lichen?

So what features can we expect from a wee investable lichen?

It should be a real thing.

An investable must be a real thing. I see among my peers a great desire to rip pixels off the screen. Put the screen on your face and call it virtual reality; put the sensors on your skin and call it embodiment; put reality back into the pixels and call it augmented reality — do something to make it real again. Lanier echoes Walter Benjamin’s construction of the aura as he expresses the lure of the object:

A real painting is a bottomless mystery, like any other real thing. An oil painting changes with time; cracks appear on its face. It has texture, odor, an a sense of presence and history. (133)

and again later as he points out a salient distinction between the digital and the physical:

[A digital object] will be a flat, mute nothing if you ask something of it that exceeds those expectations [of which aspects are important]. If you didn’t specify the weight of a digital painting in the original definition, it is not just weightless, it is less than weightless.
A physical object will be fully rich and fully real whatever you do to it.

Unlike Lanier, I do not think it inherent in digital objects that they be this way; we can have objects that are both real and digital. As ethnologist and researcher Sherry Turkle puts it, technology is ”as much an architect of our intimacies as our solitudes.” In fact, the watcher objects with whom we began are precisely this combination. But as an articulation of the goal of these objects, the insistence they be “fully rich” sets a worthwhile goal.

The richness of physical objects does not lie only in their physical presence and concomitant persistence in the world once we are gone (though we will return to those values later); there is also great value in the relationships and connections we are able to find with our objects.

In The Inner History of Devices and Evocative Objects, Turkle collects stories of the often transformative relationships between everyday people and the objects around them. There is E. Cabell Hankinson Gathman, who holds her study abroad to Japan frozen inside a cell phone and Barry, the man for whom computers open the world by making math accessible and tactile. He tells Turkle, “that with a computer and calculator, ‘The numbers are in your fingers .... They put mathematics in my hands and I am good with my hands.’” (Turkle, Devices 28) Here we see the ability of basic digital items to morph our bodies and minds without VR at all.

In her piece on the digitization of Le Corbusier’s digital archives, Susan Yee articulates the connection to the past derived from physical objects, reveling in her ability to touch the paper Le Corbusier touched and do the math he has scrawled into the margins. (Turkle, Objects 32–33) She laments the passing to the digital, “It made the drawings feel anonymous and me feel anonymous.” (35)

“The acknowledgement of the power of objects has not come easy,” says Turkle, tracing the reluctance to engage with the power of material culture to the same Western predilection for abstraction that gave us free-floating data. Fortunately, through the 1980s and into today material culture scholars located the value of objects — often found in the heart of abstraction territory (that is, science) (Turkle, Objects 6–7) — and Turkle and other scholars of material culture have collected the stories and theories behind their workings and we can build on these.

We can also build on the investigations into how physical objects can work cross-culturally to express experiences outside words. Human-computer interaction researchers Jarmo Laaksolahti, Katherine Isbister and Kristina Höök created and tested the sensual evaluation instrument (SEI), “an instrument previously developed to facilitate nonverbal self-report of emotion, which consists of eight sculpted objects.” (Laaksolahti 165) These sculptural objects, ranging from smoothly stone-like to spiky, to complexly blobby, were deployed in a number of studies, including one that used photos from the International Affective Picture Set, computer games, and conversation to test the expressive values of the objects across two cultures, Swedish and American. While some differences were observed, overall the ability of certain shapes to be read across cultures was established. (Isbister 319–322)

Working cross-culturally can also mean working outside the sorting-by-type bubble-creation common to contemporary technology. We can take an approach that supports heterogeneity and cultural cross-cutting, as exemplified in Lanier’s description of an oud forum he frequents, where all ethnicities and skill levels unite to create a rich community around the instrument. This of course is only possible by being partly digital and not constrained by geography or time zone. (Lanier 71-72)

Altogether, physical-digital objects provide the base desirable in an invested object: they can communicate across cultures, they are capable of embodying full emotional experiences, and our physical experiences with them are rich and complex.

It should be a private thing.

If information will be brought down to live again within its context in an investable object, we must consider how to avoid the pitfall of recreating the surveillance-friendliness of the watcher object. Wanting to pollute the possible means making surveillance difficult if not impossible.

Kristina Höök, who worked with Isbister on the SEI, has also researched what she calls interactional design:

An interactional perspective on design will not aim to detect a singular account of the “right” or “true” emotion of the user and tell them about it, but rather make emotional experiences available for reflection. That is, to create a representation that incorporates people’s everyday experiences that they can later reflect on. Users’ own, richer interpretation guarantees that it will be a more “true” account of what they are experiencing. (647)

The goal of designing from this perspective is to preserve user’s autonomy and privacy. Höök et al. begin by describing privacy in the negative, pointing to un-private systems, like the EmpathyBuddy email system or affective learning tools from Picard’s group, which use different methods to build up an emotional model of the student and use this to shape the learning approach the system takes. Höök describes the problem:

All of these applications regard emotion as something that can be measured, isolated and then used as a basis for how to make a system respond. This makes these kinds of systems potentially vulnerable to privacy protection issues. By that, we do not mean that the problem necessarily lies in what these systems store on the computer and whether that can be properly protected by various security solutions. Our concern lies on the level of what users may feel about systems that claims to know something about their emotional states, perhaps building profiles of them. (648)

When the system proposes to know a user’s state, particularly when it uses methods to which the user has no access, the system can be seen as violating privacy. (649)

Höök continues to address the problem of autonomy:

Overall, these systems may also threaten users autonomy since they do not hand over any control to the user, but instead decides what to communicate to others (be it friends or teachers or the system itself) about the end-user’s emotional state. (648)

To counter these issues Höök et al. advocate for designs that “assume that the meaning of an emotional process is created by people and that affective interactive systems should be such that users are encouraged to negotiate these meanings themselves.” (649) After reviewing designs they believe to meet this approach, in particular, the eMoto and Affective Diary, the authors offer up six principles based on work from Kirsten Boehner and colleagues:

  1. The interactional approach recognizes affect as a social, cultural and bodily product

  2. The interactional approach relies on and supports interpretive flexibility

  3. The interactional approach is non-reductionist

  4. The interactional approach supports an expanded range of communication acts

  5. The interactional approach focuses on people using systems to experience and understand emotions

  6. The interactional approach focuses on designing systems that stimulate reflection on and awareness of affect (652–53)

which can be summed up in three design approaches:

  • Designing open familiar surfaces that can be appropriated by users

  • Leaving the interpretation to the user through a balanced ambiguous design elements

  • Involving users in affective loop experiences (653)

Openness and balanced ambiguity are approaches that allow users to invest the system with their own meanings and communicate them in the manner they find most true and accurate. The affective loop is a process wherein the system reflects the users emotions back to them, creating both a relationship with the system and deeper understanding within the mind of a user.

If we keep the suggestions and cautions of interactional design in mind when considering the investable object, we can be sure we are working towards creating objects unlike the watchers — objects that support the privacy and autonomy of the users.

It should be a reflexive thing.

Investable objects ought to be reflexive objects instead of homeostatic ones. When Höök et al. describe the interactional point-of-view, they are also describing a reflexive one. Interactional design is design that works to communicate emotions and these:

are not only cognitive phenomena, but are also experienced as physical, bodily processes, and are in turn influenced by our bodily experiences. The way we experience emotions is shaped by the culture we live in and the specific social setting they occur in. (647–48)

Continuing, they write

From an interactional perspective, communication of emotions is not simply an information transfer problem; it is about physically and intellectually experiencing the whole range of emotions that make up a conversation. We name them affective loop experiences, experiences where it is not possible to separate the intellectual from sensual experiences, nor to single out what is my individual experience from the overall experience arising in a dialogue with a friend or in dialogue with a system. (648)

This in itself rhymes with McKay’s alternative information theory as reported by Hayles, which “[a]rguing for a song correlation between the nature of a representation and its effect, ... recognized the mutual constitution of form and content, message and recover.” (Hayles 56)

Turkle’s view of evocative objects, on which the investable object draws, also refuses the image of free-floating, decontextualized information. Evocative objects, she writes, are those considered “as companions to our emotional lives or as provocations to thought.” They “[underscore] the inseparability of thought and feeling in our relationship to things.”

We find it familiar to consider objects as useful or aesthetic, as necessities or vain indulgences. We are on less familiar ground when we consider objects as companions to our emotional lives or as provocations to thought. The notion of evocative objects brings together these two less familiar ideas, underscoring the inseparability of thought and feeling in our relationship to things. We think with the objects we love; we love the objects we think with. (Turkle, > Objects> 5)

Or as Hayles herself wrote,

If what is exactly stated can be done by a machine, the residue of the uniquely human becomes coextensive with the linguistic qualities that interfere with precise specification — ambiguity, metaphoric play, multiple encoding and allusive exchanges between one symbol system and another. (Hayles 67)

We see an expression of this sense of reflexive context and ambiguity, and of design for privacy and autonomy in the SEI’s non-anthropomorphic, non-singly mapped approach. The sculpted objects are meant to express an emotion without using faces to describe them or being mapped to one exclusively. Conflict is often represented by users with the spiky object or the anteater, but neither is said to consistently represent conflict and can be used with other objects to express multi-layered emotions. It takes on meaning culturally, in context and in conversation. The SEI is on the way to being an investable object, which is a reflexive one.

It should be something people can make themselves. (Maybe.)

As a final corollary to reflexivity, it is possible that an investable object should be something people can make themselves. If this is case, it would also be a way of polluting the possible currently proffered by the maker movement, today a defender of the real object but also a breeding ground for domesticating the watcher object via DIY sensor creation — as well as generally being a vector for capitalist propaganda.

As Evgeny Morozov relates in his history of the maker movement for the New Yorker, the flowering of DIY electronics and related products begins with Stewart Brand and his Whole Earth Catalog, itself closely tied on the one hand to the cyberneticists of the Macy Conferences and on the other to the Homebrew Computer Club of Silicon Valley, birthplace of the that shrine to disembodied information, the home computer. Brand was a great proselytizer for the hacker ethic, now co-opted so that “our hackers aren’t smashing the system; they’re fiddling with it so that they can get more work done.” He elaborates, “In this vision, it’s up to individuals to accommodate themselves to the system rather than to try to reform it. The shrinking of political imagination that accompanies such attempts at doing more with less usually goes unremarked.” (Morozov)

Brand himself has moved on to championing makers, who “take whatever we’re not supposed to take the back off of, rip the back off and get our fingers in there and mess around. That’s the old impulse of basically defying authority and of doing it your way.” (Brand, quoted in Morozov) And yet, the actual devices produced by these makers tend to be either robot toys, fun but unremarkable, or sensors, made to watch and report back on plants, heat, and other free-floating information around us. In fact, this movement is the ground from which the watcher object startups mentioned in the introduction have often sprung.

Chris Anderson, former Wired editor and current drone entrepreneur, is the maker’s Brand, cheering on the movement as a place where the best rise up with “ever-accelerating entrepreneurship and innovation with ever-dropping barriers to entry.” (Anderson, quoted in Morozov) The makers are above all capitalists, the economic theory of free-floating information and buyer preference removed from external costs and embodied repercussions. Another grand old man of the movement, TechShop’s Mark Hatch, likes to talk about how access to fabrication equipment has rehabilitated folks like “labor organizers” into entrepreneurs. (Morozov)

Morozov identifies the core of our sickness again in imagination:

Our tech imagination, to judge from catalogues like “Cool Tools,” is at its zenith. (Never before have so many had access to thermostatically warmed toilet seats.) But our institutional imagination has stalled, and with it the democratizing potential of radical technologies. We carry personal computers in our pockets — nothing could be more decentralized than this! — but have surrendered control of our data, which is stored on centralized servers, far away from our pockets.

Yet, an object made by a person to communicate and express some set of emotions, is a solidly reflexive object. It can be invested with the personal not only in terms of content but in terms of the object itself. One can even begin to imagine the use of materials that allow the observer or receiver to mark the object in response, reflecting physically the widening gyre of the reflexive system.

And if we were able to disseminate the means of creating reflexive, personal, investable objects, we might pollute the narrative currently living inside the maker movement, with its emphasis on capitalism and decontextualized information, its production of watcher objects for the profits of a few.

This characteristic allies the investable object with the goals of the FabLab, as developed by Neil Gershenfeld. The labs are “globally connected open workshops where people can meet, collaborate, interact and exchange ideas, machines, tools, materials, and software with the common purpose of making distinctive and digitally designed objects (from scratch) in an accessible and cheap way.” (Walter-Hermann and Büching 13) They were born from Gershenfeld’s experience at MIT, where a class he created for advanced physical sciences students, “How to Make (Almost) Anything,” ended up attracting a hundred interested artists, architects, and engineers. Not only was he surprised by the students, but approach to learning made a big impression: “the class turned out to be a bit of an intellectual pyramid scheme,” he writes, noting that students learned machines and techniques just-in-time then passed them on to others when others needed them. (Gershenfeld 5–7)

Unlike TechShops or other canonical maker experiences, FabLabs are not only about empowering Western entrepreneurs. They were designed to be spun up relatively cheaply and built in nontraditional locations: the inner city, Sami land. (Gershenfeld 5–7) Further investigation into the lessons of the FabLab might be able to point us towards other possibilities of a make-it-together investable.

Specific-ish II: What It Might Contain

And so the primary characteristics of the investable object itself have been sketched, but what should be invested into the object? Whether or not people make the physical container themselves, whether or not the object is reflexive, there must be personal content, communication, emotion and it must come from the object’s owner. But how do we even talk about content without inadvertently slipping into the common construction of information as unmoored entity again? Höök et al. choose to speak of emotion, using a term already grounded in subjectivity and generally accepted to be contextual, and in particular the emotion generated by interactions with games and other stimuli. Coming from a scientific perspective, this is enough — the researchers are looking for emotional-expressive tools as part of an investigation into a particular set of hypotheses around design.

In the case of attempts to pollute the possible, however, it would be ideal to develop a tool that can be used in a broader context, with a stimulus that is common but ambiguous, vast in its experience and interpretation, but known already as a continuous phenomenon, emotional, specific, embodied, a target for communication throughout human existence — for this, I submit memory.

As Walter Benjamin writes about Proust, “An experienced event is finite — at any rate confined to one sphere of experience; a remembered event is infinite, because it is only a key to everything that has happened before and after it.” (Benjamin 202)

Memory is arguably what makes us ourselves: it is our experiences that shape our ongoing interpretation of ongoing phenomena. Memory is embodied in multiple ways: we can only remember the places where we have been, physically or psychically; as we remember, we reconfigure actual synapses in our physical brain (Lehrer 75–78); we can be spurred to remember through our body, in particular via olfactory stimulus (see Herz, or really anyone who ever quoted Proust or typed the word madeline).

Memory is also deeply reflexive. It may be the ultimate system wherein the observer changes the observed and the observed changes the observer. This action can be described as a palimpsest, with each cycle adding layers, generating complex meaning, much like Proust’s drafts of A la recherche du temps perdu themselves; this action can be a bricolage, cited by Turkle as the practice that inspired her work in material culture — in her case literally as the search for her father’s memories in an assemblage of objects (Turkle, Objects 5).

Memory, social and personal, is known to live within objects. As Maurice Halbwachs writes in “Space and the Collective Memory,”

In an antique shop, the various eras and classes of a society come face to face in the scattered assortment of household belongings. One naturally wonders who would have owned such an armchair, tapestry, dishes or other necessities. Simultaneously (it is basically the same thing), one thinks about the world recognizable in all this, as if the style of furniture, the manner of decor and arrangement, were some language to be interpreted. (47)
In this way, memory as content doesn’t have to doom our project to solipsism or isolation.

Finally, memory is the domain of the artist. If the engineers have given us the tools to digitally augment our capabilities along with the sickness of information, the artist has access to “a body of theory that has considered memory’s intersection with the imagination and the aesthetic,” (Farr 18) and thereby a thoughtful human salve for the injury.

More practically, too, the art world has already lived through the return of an embodied memory. Ian Farr explains,

If, from the end of the 1950s onwards, a spectre haunted modernism, decade by decade weighing down on its structures until they collapsed into noble but now incoherent fragments, it was the spectre of memory.... The liberation from tradition and historical contingency, the presentness yet simultaneously abstracted remoteness considered central to mid-century modernism’s aesthetic aspiration was assailed the return of what it had seemingly repressed.

This return took the form of postmodern approaches to memory or, as he writes, “to be marginally less imprecise, strands of artistic practice increasingly inflected by remembrance and forgetting.” (Farr 13)

But let’s step away from lyricality for a moment and to add a little more precision to our questions; to create a prototype we must be able to state our aims a little more straightforwardly, art or not. First we must consider what we are actually discussing.

Memory can be described in almost too many ways. It can be the Bergsonian cone interacting with the plane of experience. It can be documentary, as the memory that exists frozen within a photograph, as with the work of Miroslav Tichy (Farr 12).

It can be the antithesis to this document, as Siegfried Kracauer posits in “Memory Images”: “Memory is neither the entire spatial appearance of a state of affairs nor its entire temporal course,” and so,

Since what is significant is not reducible to either merely spatial or merely temporal terms, memory images are at odds with photographic representations. From the latter’s perspective, memory images appear to be fragments — but only because photography does not encompass the meaning to which they refer and in relation to which they cease to be fragments. Similarly from the perspective of memory, photography appear as a jumble that consists partly of garbage. (Kracauer 45–46)

It can be the hallucinatory surrealist memory of Andre Breton’s Nadja. As Farr writes, connecting the work of Sartre and Breton via Paul Ricoeur,

Whether the trigger is a familiar landmark on an everyday walk suddenly made strange by a change incident of light or shadow, or a curious object or person emerging into vision for the first time, the > état d’attente> (state of expectation) of the surrealist viewer is a mode of being that allows an opens up to this semi-hallucinatory pathway in a process of memory-association. (14)

It can be a property moving from the realms of the cultural to the archival, as suggested by Pierre Nora, who locates true memory in the “spontaneous, social and all-embracing” French conception of a remembrance rooted in a place, and contrasts it with three modern versions of memory. There is archival memory, which “relies entirely on the specificity of the trace, the materiality of the vestige, the concreteness of the recording, the visibility of the image.” (62) There is memory as individual duty, the memory that creates contents to be archived, with each of us a memoirist. He also adds a third type of memory to be considered, “alienated memory.” This is a modern sort, the result of our discontinuity with true memory, wherein we scramble to recreate the past, to understand the lost country from which we are irretrievable separated. (Nora 64–66) Nora reports these developments with a sense of a loss and an eye on the absurdity of our attempts to prevent any trace from being lost — again nearly burying us in the garbage Kracauer finds in photographs.

Memory can also be plotted along the axis from private to public, or as I like to think of it, from Proust to Perec. In discussing the second type of memory, individual, Nora ties Proust (along with Freud) to the shift of memory into the psychological, “from the social to the individual, from the concrete message to its subjective representation from repetition to remembrance.” The result? “Memory becomes a private affair.” (64) (We do not need to agree with the entire theory, and I do not, to see the resonance of the description.)

In Proust, memory is intensely personal. Though it bears up its global truths through the repetition of motifs and phrases, and tells us it will from the moment Swann’s little phrase is introduced in Swann’s Way, it can do so only through the means of the narrator’s careful reporting on the events of his life.

Perec works with memories that are public, nearly dispassionate, but still need to be “provoked, rescued from oblivion.” This he does by enumerating observations in a public place, as in Attempt to Exhaust a Parisian Space, or by enumerating memories themselves as with Je Me Souviens. The memories Perec is interested in are of a particular sort as well, “memories which truly render the ’tissu du quotidien’ (fabric of everyday) — a body of experience that transcends our own individuality and yet invokes a commonality of experience.” These “cannot be purely personal (what happened to me) or factual (what happened to be the case).” (Sheringham 88)

And yet, though we can oppose Proust and Perec in terms of denotative contents: long, winding sentences entwined around I versus short, dispassionate lists on TV stars and the number of cars in the street, both work to draw information out of the aether and root it again in the particular.

When we want to target memory, do we aim to target the hallucinatory, the historical, the archival, the fragmentary and mystical? Do we want the public or private, to facilitate a Proust or a Perec? Or is this a distinction without difference? These questions, I think, can only be teased out most fully in the realm of the prototype.

Likewise, we can look into the methods of reflexive meaning production the invested object might engage with without choosing one as an untested winner. Once more using Proust, that modernist memory heavyweight, as lens, Roger Shattuck outlines three principles for generating meaning with fragments: the cinematographic, the montage, and the stereoscopic.

The first creates action and meaning by creating the sensation of motion; the second “rejects the accumulation of small differences [that is characteristic of the first] for the exploitation of larger associative or dissociative leaps that suggest the meaning of a scene or situation by contrast.” The third, the stereoscopic is the reflexive and most Proustian, arresting time to allow us to “hold contradictory aspects of things in the steady perspective of recognition, of relief in time.” (Shattuck 34–35)

“Each one develops a fundamental means of assembling the multiplicity of images or instantanés that make up our first impression of Proust’s universe — and of life itself.” (Shattuck 34) Each one might be used to guide the reflexive experience of which an invested object is part.

We might also consider how we are lost and found in reflection, or, as Briony Fer puts it, discussing Jorge Luis Borges’s Funes the Memorius, a novel wherein perfect memory becomes no memory at all, “If we are lost without repetition, we are also lost to it and in thrall to it. As the very ground of consciousness, repetition cuts both ways, both shoring up and shattering its fragile and precarious hold.” (71)

Pitfalls and Objections

And so we have an idea of an object, the historical justification for its necessity, and an idea of the type of content it might produce. But as with all objects of ambition, there are pitfalls and objections to consider.

For instance, Lanier might argue that creating an object we hope to have a relationship with, one that can contain “the residue of the uniquely human” (Hayles 67) would be expanding the “empathy circle” too far, undermining that which is special about humanity. (Lanier 40) He might also argue that the nostalgic — and when we talk about using memory we must be clear we are talking explicitly about captured and induced nostalgia — is fundamentally non-authentic and a “waste” of our new technological capabilities. (Lanier 130)

Fortunately, this argument offers little more than modernism-fueled novelty worship, and I hope the array of memory-inspired artworks available to us will reveal that modes of the past can be a rich medium through which to explore the present.

Sherry Turkle brings up a more complex pitfalls in the preface to The Inner History of Devices. She tells of Gordon Bell, computer pioneer (this is beginning to sound familiar) and creator of MyLifeBits Gordon Bell. Working with a team from Microsoft, Bell is animated by the “idea of a complete, digitally accessible life.” This means scanning all of his books, recording all of his lectures, capturing everything from notes to logos in pursuit of a total archive for their descendants. (Turkle, Devices 24) This is Pierre Nora’s nightmare, too.

Turkle contrasts this approach with two other. On the one hand is Lillian Hellman’s approach in which meaning “comes in pentimento, in the painter’s layering of paint, in his ‘repentance’ as he finds what he wants in the process of repainting.” On the other there is Turkle’s own mother, saving photographs with poems on the back in a large drawer, pulling them out to savor again, sometimes in joy, sometimes in mourning. Of the first she asks, “What will become of this kind of reworking, when, in digital culture, people’s fantasies shift from telling the story of a life to having a complete record of it?” Of the second she fears corrosion:

Of course, the digital archive is only a resource; it remains for us to take its materials as the basis for a deeply felt enterprise of recollection. But one wonders if the mere fact of the archive will not make us feel that the job is already done. (Turkle, > Devices> 24, 26–27)

In response to this, I can only push back on the idea that there is an unchangeable digital character as opposed to a contingent digital deployment. That is, the same context and reception understanding we must see in information, we can also see in the story of the digital.

Looking Around, Looking Forward

Having considered the pitfalls above, I am still committed to the usefulness of the invested object. This is an object that will aim to rescue information from its engineer-inspired, cybernetic decontextualization by polluting the possible with different ideas in action. To undertake these goals such an object:

  • should be a real thing
  • should be a private thing
  • should be a reflexive thing
  • should (maybe) be something people can make themselves.

An instance of such an object could take as its content, its provocation, memory, an experience that is as embodied, contextual, ambiguous, foundational, familiar and broad as the information we’d like to rescue.

The next steps in investigating this hypothesis, in iterating on the object and getting it into the world is to consider previous artistic works on memory and then create my own prototype of an invested object.

Given research into memory provocation from Rachel Herz demonstrating the power of the olfactory in terms of allowing people to better conjure and more acutely experience memory and that these memories were less often conjured willingly (Herz 217), the olfactory path is a promising one. It also allows us to work within the physical very explicitly, again reinforcing the embodiment of the communication. This is an experiential process and as such, should consider experiential works that have gone before.

Fluxus works like Ay-O’s Finger Box and Smell Chess highlight the senses of touch and smell, creating an aesthetic experience through interaction with the physical world. The latter is a game of chess in which the pieces are each a vial of spices and must be smelled to played or understood, whereas the former comprises a set of identical boxes, each of which contains a touchable item, from sponges to cotton balls to a nail, and thus must be approached carefully, reverently. As Hannah Higgins describes it,

Cutaneous information is gained by direct contact with materials. Apart from the lips and tongue, the index fingertip is the most sensitive cutaneous organ and is therefore particularly well suited to use with the > Finger Box> . When users plunge a finger into the box, their curiosity has overcome the sense of fear inherent in exploring the unknown. That several > Finger Boxes> contained nails indicates Ay-O’s determination not to sidestep the challenge the work could issue: the danger to the instinctively apprehensive, hesitant user, who must touch the box, but carefully, with an “enquiring, learning gesture.”
> By requiring users to handle them gently, the boxes set up the potential for non-destructive knowing> . (Higgins 26, emphasis mine)

At the end, Higgins identifies the subversive potential of Fluxus works, another venue to show a kind of information that is enriched by its consumption and not destroyed.

Both works give life to the theories of experiential art from John Dewey that underlie the Fluxus movement:

In art as an experience, actuality and possibility or ideality, the new and the old, objective material and the personal response, the individual and the universal, surface and depth, sense and meaning, are > integrated into an experience> in which they are all transfigured from the significance that belongs to them when isolated in reflection. (John Dewey, quoted in Higgins 31–33)

Also relevant is the work of Brazilian artist Cildo Meireles, who creates “art concerned with the senses and sensorial consciousness” as part of Brazil’s Neo-concrete movement, “where the body can be seen as the location of a work of art.” (Meireles 221) Of particular interest is the piece Volatile, where a candle and the scent of gas work together to create the “sensation of danger” in the viewer. In this case a complex feeling is evoked via the combination of scent and sight.

Beyond the experiential, there are a number of other artists of memory to consider. As mentioned before, Proust, Perec, and Borges each has developed complex works around a theory of memory and meaning. In the visual arts, Robert Morris created Anti-forms, works that “worked to ‘forget’ their form.” (Morris 93) Pierre Huyghe’s The Third Memory looks at the bank robbery in Dog Day Afternoon through the film and interviews with the actual robber.

Idris Khan creates palimpsests, focusing on repetition and variation through multiple layers, like memories themselves.

Those layers could be pictures or text or musical scores; the process of layering works wherever this is a repeating pattern with variation. But Khan is careful to dose his superimpressions just so: you can see layers but you often can’t decipher them. Each layer flavours the next but none is the right one.... There is no top layer. (Hodgson 94)

This investigation too lacks a top, for now. Though this paper has stated the case for an invested object, these lists are only a nod at the cultural context in which they must live. More research into the creation and reception of the art preceding these ideas will be required for a worthwhile final prototype to emerge.

In addition, greater research into what we know about material culture interpretations of objects is also needed. This, particularly when combined with research from Höök and Isbister, will allow us to create artifacts that are legible across time and culture.

And so forward, into the future of the past.



Cited and Background Sources

Ada, Michael. Dominance by Design. Cambridge: Harvard University Press, 2009. Print.

Benjamin, Walter. Illuminations. New York: Pantheon Books, 1968. Print.

Farr, Ian, ed. Memory. Cambridge: MIT Press, 2012. Print.

Fer, Briony. “The Infinite Line.” Farr 71–75.

Gershenfeld, Neil. Fab. New York: Basic Books, 2008. Print.

Gertner, Jon. The Idea Factory: Bell Labs and the Great Age of American Innovation. New York: Penguin Books, 2013.

Halbwachs, Maurice. “Space and the Collective Memory.” Farr 47–49.

Hayles, N. Katherine. How We Became Posthuman. Chicago: University of Chicago Press, 1999. Print.

Herz, Rachel. “A Naturalistic Analysis of Autobiographical Memories Triggered by Olfactory Visual and Auditory Stimuli.” Chem. Senses 29: 217–224, 2004. Web.

Higgins, Hannah. Fluxus Experience. Berkeley: University of California Press, 2002. Print.

Hodgson, Francis. “Idris Khan.” Farr 94–96.

Höök, Kristina, Anna Ståhl, Petra Sundström, Jarmo Laaksolahti “Interactional Empowerment.” CHI 2008 Proceedings: Dignity in Design: 647–656, 2008. Web.

Isbister, Katherine, Kia Höök, Jarmo Laaksolahti, Michael Sharp. “The sensual evaluation instrument: Developing a trans-cultural self-report measure of affect,” International Journal of Human-Computer Studies 65: 315–328, 2007. Web.

Kracauer, Siegfried. “Memory Images.” Farr 45–47.

Laaksolahti,Jarmo, Katherine Isbister and Kristina Höök. “Using the Sensual Evaluation Instrument.” Digital Creativity 20:3, 165–175, 2009. Web.

Jonah Lehrer, Proust Was a Neuroscientist. New York: Mariner Books, 2008. Print.

Meireles, Cildo and Charles Merewether. “Memory of the Senses.” Grand Street 64: 216–223, Spring 1998. Web.

Morris, Robert. “In Conversation with W.J.T. Morris.” Farr 92–94.

Murray, Janet H. “Inventing the Medium.” The New Media Reader. Noah Wardrip-Fruin and Nick Montfort, eds. Cambrige: MIT Press, 2003. 3–11. Print.

Nora, Pierre. “Realms of Memory.” Farr 61–66.

Petersen, Anne Helen, “Big Mother Is Watching You,” Buzzfeed: January 1, 2015. Web.

Proust, Marcel. Swann’s Way. Trans. Lydia Davis. New York: Viking, 2003. Print.

Sheringham, Michael. “Perec: Uncovering the Infra-Ordinary.” Farr 86–92.

Shattuck, Roger. “Proust’s Binoculars.” Farr 30–41.

Turkle, Sherry. Inner History of Devices. Cambridge: MIT Press, 2008. Print.

—————. Evocative Objects. Cambridge: MIT Press, 2007. Print.

Walter-Hermann, Julia and Corinne Büching. Fablab: Of Makers, Machines, and Inventors. Bielefed: Transcript, 2013. Print.