Making in Mixed Reality

Since Microsoft and Magic Leap announced their respective headset technologies in 2017, there has been a lot of hype surrounding Mixed Reality. The ability to overlay human-readable information over a visible experience of the physical world provides a more versatile connection between the digital and physical than our current screen interfaces. I recently tried the Hololens in a workshop organised to demonstrate its use in hand making material prototypes. The workshop was hosted by Eduardo Barata and Dagmar Reinhart at Sydney University Architecture school and was enabled by the Fologram team and their software.

Mixed reality differs slightly from Augmented Reality — AR (virtual objects placed within a pixelated representation of reality) and varies a lot from Virtual Reality — VR (a complete virtual image) by projecting an image onto a transparent screen in front of your eyes. The effect is natural on the eyes as the extra information is added to the typical view of the world. For architecture this has many benefits;

Architectural Representation

Mixed reality coupled with a three-dimensional model can project virtual geometry into space, the effect is a new type of architectural representation. Instead of describing architecture through flat drawings or pixelated screens, a virtual form becomes part of the visual aspect and adjusts the position of the head in space. The overlay of three-dimensional digital information provides a real-time relationship between the physical context and the digital model and produces a 3d experience rather than just a 3d representation. From the perspective of architecture, a 3d and therefore spatial representation is beneficial to accurately show where to place and join materials into a structure and to present an experience of architecture before it has been materialised. What is more, many Hololens headsets can exist in the same model meaning multiple people can join in the construction or experience.

Multimodal design interaction

Once this pipeline of Hololens communicating with Fologram (running within Rhino and Grasshopper) has been established, opportunities for innovation exist in the way users can interact with the model in space through gestures, and through how geometry is processed within the Rhino and Grasshopper environments. The passing of data between the digital and physical environments through the Hololens creates a cyber-human-physical system. A possible scenario for this is a human response to geometry in space, an element is adjusted via a machine recognisable gesture, which subsequently recalculates the virtual geometry within a parametric model, then feeds back into the virtual overlay. The outcome here is that within a design environment set up by the Hololens and Fologram virtual and material form can co-emerge through human aesthetic decisions, and computed material behaviours.

Construction Complexity

Applications of the Fologram system are already being explored through university design studios and construction projects. In such experiments, mixed reality has enabled faster or more complex structures to be achieved through the externalisation of information onto the context. The 1:1 virtual overlay of a dynamic digital model means that tasks previously requiring high levels of tacit material knowledge and craft skill can be done by anyone with hands and arms.

Material Outcomes

In the marketing communication of Fologram, the most significant benefit from mixed reality is the quality of material outcomes coupled with a reduction in time for the humans in the cybernetic loop. The promise of increased material products and reduction of time from increased efficiency in energy expenditure reminds me of the Taylorist promise of early twentieth century mechanical production.

The potential of mixed reality is for a greater balance in human action and computational calculation which helps to reconcile human craft with the digital that has come to alienate the more tactile based architectural practitioners. However the assemblage of material through procedural information rather than cognitive processes removes an experience from the human for the benefit of the object, this needs considering more critically. What’s more the transfer of ability onto multiple humans through information introduces a political imbalance between the producer of the digital system and those tasked with its assembly. Both of these issues will be considered in more depth in the second half of this post.

‍