Augmented and virtual reality

Subterms

    Latest story

    113 Shares119 Views

    Co-creating climate futures with real-time data and spatial storytelling

    by

    Virtual story worlds and game engines aren’t just for video games anymore. They are now tools for scientists and storytellers to digitally twin existing physical spaces and then turn them into vessels to dream up speculative climate stories and build collective designs of the future. That’s the theory and practice behind the MIT WORLDING initiative.

    Twice this year, WORLDING matched world-class climate story teams working in XR (extended reality) with relevant labs and researchers across MIT. One global group returned for a virtual gathering online in partnership with Unity for Humanity, while another met for one weekend in person, hosted at the MIT Media Lab.

    “We are witnessing the birth of an emergent field that fuses climate science, urban planning, real-time 3D engines, nonfiction storytelling, and speculative fiction, and it is all fueled by the urgency of the climate crises,” says Katerina Cizek, lead designer of the WORLDING initiative at the Co-Creation Studio of MIT Open Documentary Lab. “Interdisciplinary teams are forming and blossoming around the planet to collectively imagine and tell stories of healthy, livable worlds in virtual 3D spaces and then finding direct ways to translate that back to earth, literally.”

    At this year’s virtual version of WORLDING, five multidisciplinary teams were selected from an open call. In a week-long series of research and development gatherings, the teams met with MIT scientists, staff, fellows, students, and graduates, as well as other leading figures in the field. Guests ranged from curators at film festivals such as Sundance and Venice, climate policy specialists, and award-winning media creators to software engineers and renowned Earth and atmosphere scientists. The teams heard from MIT scholars in diverse domains, including geomorphology, urban planning as acts of democracy, and climate researchers at MIT Media Lab.

    Mapping climate data

    “We are measuring the Earth’s environment in increasingly data-driven ways. Hundreds of terabytes of data are taken every day about our planet in order to study the Earth as a holistic system, so we can address key questions about global climate change,” explains Rachel Connolly, an MIT Media Lab research scientist focused in the “Future Worlds” research theme, in a talk to the group. “Why is this important for your work and storytelling in general? Having the capacity to understand and leverage this data is critical for those who wish to design for and successfully operate in the dynamic Earth environment.”

    Making sense of billions of data points was a key theme during this year’s sessions. In another talk, Taylor Perron, an MIT professor of Earth, atmospheric and planetary sciences, shared how his team uses computational modeling combined with many other scientific processes to better understand how geology, climate, and life intertwine to shape the surfaces of Earth and other planets. His work resonated with one WORLDING team in particular, one aiming to digitally reconstruct the pre-Hispanic Lake Texcoco — where current day Mexico City is now situated — as a way to contrast and examine the region’s current water crisis.

    Democratizing the future

    While WORLDING approaches rely on rigorous science and the interrogation of large datasets, they are also founded on democratizing community-led approaches.

    MIT Department of Urban Studies and Planning graduate Lafayette Cruise MCP ’19 met with the teams to discuss how he moved his own practice as a trained urban planner to include a futurist component involving participatory methods. “I felt we were asking the same limited questions in regards to the future we were wanting to produce. We’re very limited, very constrained, as to whose values and comforts are being centered. There are so many possibilities for how the future could be.”

    Scaling to reach billions

    This work scales from the very local to massive global populations. Climate policymakers are concerned with reaching billions of people in the line of fire. “We have a goal to reach 1 billion people with climate resilience solutions,” says Nidhi Upadhyaya, deputy director at Atlantic Council’s Adrienne Arsht-Rockefeller Foundation Resilience Center. To get that reach, Upadhyaya is turning to games. “There are 3.3 billion-plus people playing video games across the world. Half of these players are women. This industry is worth $300 billion. Africa is currently among the fastest-growing gaming markets in the world, and 55 percent of the global players are in the Asia Pacific region.” She reminded the group that this conversation is about policy and how formats of mass communication can be used for policymaking, bringing about change, changing behavior, and creating empathy within audiences.

    Socially engaged game development is also connected to education at Unity Technologies, a game engine company. “We brought together our education and social impact work because we really see it as a critical flywheel for our business,” said Jessica Lindl, vice president and global head of social impact/education at Unity Technologies, in the opening talk of WORLDING. “We upscale about 900,000 students, in university and high school programs around the world, and about 800,000 adults who are actively learning and reskilling and upskilling in Unity. Ultimately resulting in our mission of the ‘world is a better place with more creators in it,’ millions of creators who reach billions of consumers — telling the world stories, and fostering a more inclusive, sustainable, and equitable world.”

    Access to these technologies is key, especially the hardware. “Accessibility has been missing in XR,” explains Reginé Gilbert, who studies and teaches accessibility and disability in user experience design at New York University. “XR is being used in artificial intelligence, assistive technology, business, retail, communications, education, empathy, entertainment, recreation, events, gaming, health, rehabilitation meetings, navigation, therapy, training, video programming, virtual assistance wayfinding, and so many other uses. This is a fun fact for folks: 97.8 percent of the world hasn’t tried VR [virtual reality] yet, actually.”

    Meanwhile, new hardware is on its way. The WORLDING group got early insights into the highly anticipated Apple Vision Pro headset, which promises to integrate many forms of XR and personal computing in one device. “They’re really pushing this kind of pass-through or mixed reality,” said Dan Miller, a Unity engineer on the poly spatial team, collaborating with Apple, who described the experience of the device as “You are viewing the real world. You’re pulling up windows, you’re interacting with content. It’s a kind of spatial computing device where you have multiple apps open, whether it’s your email client next to your messaging client with a 3D game in the middle. You’re interacting with all these things in the same space and at different times.”

    “WORLDING combines our passion for social-impact storytelling and incredible innovative storytelling,” said Paisley Smith of the Unity for Humanity Program at Unity Technologies. She added, “This is an opportunity for creators to incubate their game-changing projects and connect with experts across climate, story, and technology.”

    Meeting at MIT

    In a new in-person iteration of WORLDING this year, organizers collaborated closely with Connolly at the MIT Media Lab to co-design an in-person weekend conference Oct. 25 – Nov. 7 with 45 scholars and professionals who visualize climate data at NASA, the National Oceanic and Atmospheric Administration, planetariums, and museums across the United States.

    A participant said of the event, “An incredible workshop that had had a profound effect on my understanding of climate data storytelling and how to combine different components together for a more [holistic] solution.”

    “With this gathering under our new Future Worlds banner,” says Dava Newman, director of the MIT Media Lab and Apollo Program Professor of Astronautics chair, “the Media Lab seeks to affect human behavior and help societies everywhere to improve life here on Earth and in worlds beyond, so that all — the sentient, natural, and cosmic — worlds may flourish.” 

    “WORLDING’s virtual-only component has been our biggest strength because it has enabled a true, international cohort to gather, build, and create together. But this year, an in-person version showed broader opportunities that spatial interactivity generates — informal Q&As, physical worksheets, and larger-scale ideation, all leading to deeper trust-building,” says WORLDING producer Srushti Kamat SM ’23.

    The future and potential of WORLDING lies in the ongoing dialogue between the virtual and physical, both in the work itself and in the format of the workshops. More

    More stories

    • 100 Shares179 Views

      in Environment

      Q&A: A high-tech take on Wagner’s “Parsifal” opera

      by

      The world-famous Bayreuth Festival in Germany, annually centered around the works of composer Richard Wagner, launched this summer on July 25 with a production that has been making headlines. Director Jay Scheib, an MIT faculty member, has created a version of Wagner’s celebrated opera “Parsifal” that is set in an apocalyptic future (rather than the original Medieval past), and uses augmented reality headset technology for a portion of the audience, among other visual effects. People using the headsets see hundreds of additional visuals, from fast-moving clouds to arrows being shot at them. The AR portion of the production was developed through a team led by designer and MIT Technical Instructor Joshua Higgason.

      The new “Parsifal” has engendered extensive media attention and discussion among opera followers and the viewing public. Five years in the making, it was developed with the encouragement of Bayreuth Festival general manager Katharina Wagner, Richard Wagner’s great-granddaughter. The production runs until Aug. 27, and can also be streamed on Stage+. Scheib, the Class of 1949 Professor in MIT’s Music and Theater Arts program, recently talked to MIT News about the project from Bayreuth.

      Q: Your production of “Parsifal” led off this year’s entire Bayreuth festival. How’s it going?

      A: From my point of view it’s going quite swimmingly. The leading German opera critics and the audiences have been super-supportive and Bayreuth makes it possible for a work to evolve … Given the complexity of the technical challenge of making an AR project function in an opera house, the bar was so high, it was a difficult challenge, and we’re really happy we found a way forward, a way to make it work, and a way to make it fit into an artistic process. I feel great.

      Q: You offer a new interpretation of “Parsifal,” and a new setting for it. What is it, and why did you choose to interpret it this way?

      A: One of the main themes in “Parsifal” is that the long-time king of this holy grail cult is wounded, and his wound will not heal. [With that in mind], we looked at what the world was like when the opera premiered in the late 19th century, around the time of what was known as the Great African Scramble, when Europe re-drew the map of Africa, largely based on resources, including mineral resources.

      Cobalt remains [the focus of] dirty mining practices in the Democratic Republic of Congo, and is a requirement for a lot of our electronic objects, in particular batteries. There are also these massive copper deposits discovered under a Buddhist temple in Afghanistan, and lithium under a sacred site in Nevada. We face an intense challenge in climate change, and the predictions are not good. Some of our solutions like electric cars require these materials, so they’re only solutions for some people, while others suffer [where minerals are being mined]. We started thinking about how wounds never heal, and when the prospect of creating a better world opens new wounds in other communities. … That became a theme. It also comes out of the time when we were making it, when Covid happened and George Floyd was murdered, which created an opportunity in the U.S. to start speaking very openly about wounds that have not healed.

      We set it in a largely post-human environment, where we didn’t succeed, and everything has collapsed. In the third act, there’s derelict mining equipment, and the holy water is this energy-giving force, but in fact it’s this lithium-ion pool, which gives us energy and then poisons us. That’s the theme we created.

      Q: What were your goals about integrating the AR technology into the opera, and how did you achieve that?

      A: First, I was working with my collaborator Joshua Higgason. No one had ever really done this before, so we just started researching whether it was possible. And most of the people we talked to said, “Don’t do it. It’s just not going to work.” Having always been a daredevil at heart, I was like, “Oh, come on, we can figure this out.”

      We were diligent in exploring the possibilities. We made multiple trips to Bayreuth and made these milimeter-accurate laser scans of the auditorium and the stage. We built a variety of models to see how to make AR work in a large environment, where 2,000 headsets could respond simultaneously. We built a team of animators and developers and programmers and designers, from Portugal to Cambridge to New York to Hungary, the UK, and a group in Germany. Josh led this team, and they got after it, but it took us the better part of two years to make it possible for an audience, some of whom don’t really use smartphones, to put on an AR headset and have it just work.

      I can’t even believe we did this. But it’s working.

      Q: In opera there’s hopefully a productive tension between tradition and innovation. How do you think about that when it comes to Wagner at Bayreuth?

      A: Innovation is the tradition at Bayreuth. Musically and scenographically. “Parsifal” was composed for this particular opera house, and I’m incredibly respectful of what this event is made for. We are trying to create a balanced and unified experience, between the scenic design and the AR and the lighting and the costume design, and create perfect moments of convergence where you really lose yourself in the environment. I believe wholly in the production and the performers are extraordinary. Truly, truly, truly extraordinary.

      Q: People have been focused on the issue of bringing AR to Bayreuth, but what has Bayreuth brought to you as a director?

      A: Working in Bayreuth has been an incredible experience. The level of intellectual integrity among the technicians is extraordinary. The amount of care and patience and curiosity and expertise in Bayreuth is off the charts. This community of artists is the greatest. … People come here because it’s an incredible meeting of the minds, and for that I’m immensely filled with gratitude every day I come into the rehearsal room. The conductor, Pablo Heras-Casado, and I have been working on this for several years. And the music is still first. We’re setting up technology not to overtake the music, but to support it, and visually amplify it.

      It must be said that Katharina Wagner has been one of the most powerfully supportive artistic directors I have ever worked with. I find it inspiring to witness her tenacity and vision in seeing all of this through, despite the hurdles. It’s been a great collaboration. That’s the essence: great collaboration. More

    • 138 Shares199 Views

      in Environment

      Using game engines and “twins” to co-create stories of climate futures

      by

      Imagine entering a 3D virtual story world that’s a digital twin of an existing physical space but also doubles as a vessel to dream up speculative climate stories and collective designs. Then, those imagined worlds are translated back into concrete plans for our physical spaces.

      Five multidisciplinary teams recently convened at MIT — virtually — for the inaugural WORLDING workshop. In a weeklong series of research and development gatherings, the teams met with MIT scientists, staff, fellows, students and graduates as well as other leading figures in the field. The theme of the gathering was “story, space, climate, and game engines.”

      “WORLDING illustrates the emergence of an entirely new field that fuses urban planning, climate science, real-time 3D engines, nonfiction storytelling, and speculative fiction,” says Katerina Cizek, lead designer of the workshop at Co-Creation Studio, MIT Open Documentary Lab. “And co-creation is at the core of this field that allows for collective, democratic, scientific and artistic processes.” The research workshop was organized by the studio in partnership with Unity Software.

      The WORLDING teams met with MIT scholars to discuss diverse domains, from the decolonization of board games, to urban planning as acts of democracy, to behind the scenes of a flagship MIT Climate Challenge project.

      “Climate is really a whole-world initiative,” said Noelle Selin, an MIT atmospheric chemistry professor, in a talk at WORLDING. Selin co-leads an MIT initiative that is digitally twinning the Earth to harness enormous volumes of data for improved climate projections and put these models into the hands of diverse communities and stakeholders.

      “Digital twinning” is a growth market for the game engine industry, in verticals such as manufacturing, architecture, finance, and medicine. “Digital twinning gives teams the power to ideate,” said Elizabeth Baron, a senior manager of enterprise solutions at Unity in her talk at WORLDING. “You can look at many things that maybe aren’t even possible to produce. But you’re the resource. Impact is very low, but the creativity aspect is very high.”

      That’s where the story and media experts come in. “Now, more than ever, we need to forge shared narratives about the world that we live in today and the world that we want to build for the future. Technology can help us visualize and communicate those worlds,” says Marina Psaros MCP ’06, head of sustainability at Unity, lead on WORLDING at Unity, and a graduate of the MIT Department of Urban Studies and Planning.

      In his talk on the short history of WORLDING, media scholar William Uricchio, MIT professor of comparative media studies and founder of the Open Documentary Lab, suggested that story and space come together in these projects that create new ways of knowing. “Story is always a representation,” he says. “It’s got a fixity and coherence to it, and play is — and, I would argue, worlds are —  all about simulation. Simulation in the case of digital twinning is capable of generating countless stories. It’s play as a story-generator, but in the service of envisioning a pluralistic and malleable future.”

      Fixed dominant narratives and game mechanics that underpin board games have been historically violent and unjust, says MIT Game Lab scholar Mikael Jakkobson, who shared findings for his upcoming book on the subject with the cohort. He argues that board games are built on underlying ideas of  “exploration, expansion, exploitation, and extermination. And, as it happens, those are also good ways of thinking about the mechanics of Western colonialism.”

      To counter these hegemonic mechanics and come up with new systems, community is vital, and urban planning is a discipline that plays a huge role in the translation of space, story, and democracy. Ceasar MacDowell, an MIT professor of the practice of civic design, told the WORLDING cohort that urban planning needs to expand its notion of authorship. He is working on systems (from his current position at the Media Lab) that not only engage the community in conversations but also prompt “the people who have been in conversations to actually make sense of them, do the meaning-making themselves, not to have external people interpret them.” These become dynamic layers of both representation and simulation that are not, as Uricchio suggests, fixed. 

      USAID Chief Climate Officer Gillian Calwell visited the group with both sharp warnings and warm enthusiasm: “When it comes to climate, this world isn’t working so well for us; we better start envisioning the new ones, and fast … We don’t have time to convince people that this is happening anymore. Nor do we need to. I think most of the world is having the hands-on, up-close-and-personal experience with the fact that these impacts are coming faster and more furiously than even the scientists had predicted. But one thing we do need help with on a more hopeful note is visualizing how the world could be different.”

      The WORLDING workshop is designed and inspired by the ideas and practices charted in the Co-Creation Studio’s new MIT Press book, “Collective Wisdom: Co-Creation Media for Equity and Justice,” which insists that “No one person, organization, or discipline can determine all the answers alone.”

      The five multidisciplinary teams in this first WORLDING cohort were diverse in approach, technology, and geography. For example, one is an Indigenous-led, land-based, site-specific digital installation that seeks to envision a future in which, once again, the great herds of buffalo walk freely. Another team is creating 3D-modeled biome kits of the water systems in the drought-stricken American West, animated by interviews and data from the communities living there. Yet another team is digitally twinning and then re-imagining a sustainable future in the year 2180 for a multi-player virtual reality game in a Yawanawà Shukuvena Village in the rainforests of Brazil.

      “While our workshop design was focused on developing and researching these incredible, interdisciplinary projects, we also hope that WORLDING can set an example for similar initiatives across global sectors where distances and varied expertise are not limitations but opportunities to learn from one another,” says Srushti Kamat, WORLDING producer and MIT creative media studies/writing grad.

      Most of the talks and presentations from the WORLDING workshop are available as archived videos at cocreationstudio.mit.edu/worlding-videos. More