Isolation & Gaming: The Escapes of Leveling Up in the Virtual World

Hello, we are from the ArtSciLab’s Esports Player Development (EPD) team! We understand that while gaming culture has acquired a stigma of sorts over time, it has also served as a healthy escape and a source of interaction in dire times such as the COVID-19 pandemic. Two of our members, Semra Zamurad (Research Fellow) and Lauren Bernal (Project Manager), who both had a B.S. in Psychology, have written this blog as examples of how COVID-19 has impacted their lives and how gaming allowed them to restore mental health and their social relationships.

Semra is a co-design and research assistant on the EPD team that graduated from Texas Woman’s University in May of 2019. An avid gamer herself, she spends much of her free time playing JRPGs (Japanese Role-Playing Games) and watching streams. Her relationship with gaming started when she was young, and while it has changed plenty over time, she admits that nothing quite prepared her for the way it would be impacted by COVID-19 and how that would help her relationships continue to flourish in spite of the pandemic.

Gaming has always been a part of my life, serving as the happy medium between TV shows I wish I could play a part in and books that were thrilling but not as engaging. Such games gave me the chance to play an active role in stories that were meant to help my character realize their true potential, something I struggled to do in my real life, especially while growing up. Eventually though, I did grow up and out of that mentality, and as a result, games became less of an escape from my reality and more of a fun pastime whenever I was not busy. I still quite enjoyed the plot twists and gameplay, but I was no longer dependent on it to build my confidence in real life, something I do believe was a positive development. I had grown to combat negative emotions by going to my favorite boba cafe or rendezvousing with friends at the mall. However, with the drastic shift in lifestyle caused by stay-at-home orders and safety precautions brought on by the onset of COVID-19, I suddenly found my hands tied, and for the first time in a while, I had too much time and not enough to do. In an attempt to fill my days with more than napping and despairing about the situation, I once again found solace in gaming.

It is important that I mention, though, that gaming felt very different now. As I live at home with my parents, I was so used to complaints from my family about how much time I spent playing and how that time could be better spent on other activities. However, as of late, the home has been fairly quiet; even more surprising is how often different members of the family will come into the room I am playing in to ask about how it is going and what the story is, conversations that have seldom taken place before because there was little interest for gaming on my family’s part outside of myself. It is certainly a welcome reaction, but shocking nonetheless.

And it is not just their views on gaming that have seemed to change; I have noticed it in myself and my friends as well. I am frequently engaging with friends via discord, often joining in general calls to converse at various times in the day. Many of my friends are from all over the world, but we share common interests such as gaming; the divide comes in the form of the types of games we prefer to play. Interestingly enough, in an effort to move the conversation away from the constant gloom brought on by the state of the world, game suggestions, invitations, and streams have become a daily occurrence. Many of us have become much more open-minded about the series and the types of games we normally play in an effort to connect with each other more; as new releases continue to happen on time, more and more of us have started playing the same games, which often come with online connection features that allow us to play with each other in real time. Fire Emblem: Three Houses, Persona 5, and Animal Crossing have become household names, and we continue to play outside of discord calls via iMessage games, effectively closing the mental and emotional distance between us. And still, we endeavor to find new games to play, bringing new things to talk about, new characters and stories to share, more advice to exchange, and discussions about strategies to implement.

The feeling of being active in a virtual world with different circumstances is contagious and has lifted our spirits immensely, and I believe it has a lot to do with embodied cognition. In many of these games, we each play a character that is given choices on how to respond to the world around them, and we are encouraged to respond in the way we would if we found ourselves in the same situation. Seeing those choices bring about change and ultimately progress the story rewards the players themselves by giving them a sense of hope. Each plot presents obstacles that often seem insurmountable at the beginning, but little by little, the player earns experience, builds rapport with other characters, and takes on challenge after challenge to ultimately succeed against all odds, and although it is the character in the game that is executing each action, it is the player who thoughtfully plans out each move and sees the fruits of their efforts.

Most games, if not all, are designed to give one a sense of hope and accomplishment, something that is missing from many of our lives currently as we guess and check new ways of surviving all that COVID-19 has thrown at us. With successful attempts to ride out this pandemic being far and few in between, it makes sense that many feel hopeless and trapped. But so did our characters at the beginning of each game; they too had to seek support from others, just as we rely on each other now; they too faced a long and arduous journey that tested their resolve, just as ours is being tested day by day; and they still succeeded, just as we will. If we can survive what these games put us through, we can get through this as well.

Lauren is the Project Manager for EPD. She received her B.S. in Psychology May of 2019 from UT Dallas’ Behavioral & Brain Sciences school. Her background in research is diversified from Music Perception in Cognition to Couples’ Daily Lives. She joined UTD’s ArtSciLab January 2019 and has launched this project since March of 2019. With the help of her EPD team, she has been able to lead its emergence. The EPD team worked together to publish an Annotated Bibliography: The Emergence of Esports (PDF available for download) and the 2nd Edition to be published June 12th, a recorded an interview with the ArtSciLab’s podcast Creative Disturbance: Bold Roast, launched its website www.esports-epd.com, and submitting grant proposals with UTD’s Center for BrainHealth, Center for Vital Longevity, Mavs Gaming, and Complexity Gaming. Though she mostly handles managerial tasks for the project, she recognizes the increased acceptance of gaming in times of the pandemic and the affects it has made on her team.

Personally, I have felt lethargic, apathetic, and depressed over the past few weeks. It (COVID-19) has affected my regular work schedule by making my hours less consistent than usual. On the other hand, it has helped me to see how important EPD is. Our pitch: “How do we reform the 21st century sport in a way that does not harm the players mentally or physically?” really holds during this time of crisis. Some differences I have noticed it has made for my team and being a Project Manager: 1) We have been able to stay well connected on professional and personal levels, and 2) As a PM in the UTD ArtSciLab, it has strengthened my trust in the members for my team to deliver and take initiative on their ideas. I have been thoroughly impressed by their commitment to affirm our goals with EPD.

In a sociocultural perspective, I have seen gaming bring people closer together. With confinement, virtual realities are all the more enticing. Members on the EPD team (Semra, Peter, Kristen, and Cris) have each reacted differently to the pandemic with what they can produce within EPD and online. For instance, Kristen has started streaming her gameplays on Twitch.tv, Cris has created our EPD website, www.esports-pd.com, and Semra and Peter have been more engaged with esports communities than before by communicating with friends through Discord servers and following streams.

The community in general has shown impressive acts for adaptation. One example of this is a professor used Animal Crossing: New Horizons to create a simulation of his classroom (post shared from Instagram, but message from professor was communicated through Discord). The students were offered the opportunity to journey with their avatar to the professor’s land, sit in the classroom, while they took a real-life exam. Another example of how people are exercising their creative abilities is using Minecraft for virtual dates! Love is in the air, even when we have to wear face masks, so why not indulge in a novel way to meet someone?

Times are hard right now, but we will persevere – one simulation at a time.

The Importance of User Design in Games for Health

by Maisha Razzaque

Looking to regulate your sleeping habits? Searching for a way to teach sexual health? There’s an app for that. In today’s web-based world, games for health are rocketing in popularity. These “serious” games are specifically designed to encourage behavior changes to treat a health threat. Naturally, we are inclined to ask about the validity of the games: do they work the way they were designed to work? Can serious games be used to improve health outcomes? However, we may not consider the important role of user experience — how easy and pleasant the game is — factoring into the game’s influence. Paying attention to how a game is designed and what human interaction factors considered during its development may hold the key to the future of health-based games.

Gamification in a Nutshell

Using games to affect change in real life isn’t a new concept. Educational games have been a prominent feature in the integration of technology and grade school learning since the early days of funbrain.com and Mavis Beacon. These games use the theoretical approach of teaching and testing content in small quantities — having student pass a level before moving on to the next one. The late 90s birthed the exergaming (exercise + gaming) industry — utilizing movement tracking and virtual reality to turn movement into play. In 2013, we saw the integration of health metrics, heart rate and a pedometer, into these “exergames” with Nintendo’s Wii consoles. The most visible case of implementing health goals in game design today can be found in trending augmented reality apps like Zombies, Run! and Pokémon GO. One study assessed the walking and sedentary habits of young adults before and after downloading the game Pokémon GO. The GPS-based game requires players to use their phones to search for virtual Pokémon characters as they walk through real-world locations. They found that Pokémon Go was associated with increased walking and decreased sedentary behavior. Some unexpected negative side effects of a semi-virtual game that the experimenters found included the dangers in the environment as the user is walking right into it, too immersed in his/her phone to notice! This is an example of a real-life “bug” that needs to be addressed in these mobile exergames by the developers of the programs. Perhaps the lesson here is that the health benefits of resulting increased physical behavior can only be a priority if the safety of the user in any potential semi-virtual game is accounted for by the designer. After all, what good is an app that raises your heart rate and encourages exercise if the trade-off is mistakenly walking off a cliff?

Psychological Models in Games

Encouraging health-positive behavior and tracking metrics are a great start. However, to delve deeper, we need to start at the conceptual design stage of the game. The question of gamification pivots from “does the existing design work” to “how do we design it to make sure it works?” This is where experts toy with the idea of implementing psychological models of healing in health game design—specifically, the Health Belief Model.

According to the Health Belief model, an individual’s intention to “engage in health behavior” — this includes both positive and negative behavior — can be determined by their perception of their own vulnerability to health threats and consequences. In more technical terms, the user will behave based on self-perceived strengths and weaknesses. So how do you go about using the health belief model in conceptual design: in serious games that use role-playing and sci-fi/action themes to encourage diabetes management, users are rewarded when health landmarks are met.  This offers incentive for health record integration in the game — all to ensure that metric and therapeutic goals are accomplished in the process of playing the game.

Researchers at Johns Hopkins examined design principles of serious games (for patients with chronic illnesses) based on the Health Belief Model and their influence on the games’ effectiveness in health outcomes. Adolescent participants were recruited for user experience evaluation of the games, and they found that implementing the Health Belief Model in healthcare game design increases usability in games, improving the efficacy as a health tools. But that’s just one model. In the context of future experimental design for other chronic illnesses, it’s important to gauge the value of implementing appropriate models when designing games. However, if there’s one thing that can be taken away from this study, it’s that it sets an essential precedent. If our end goal is to improve health outcomes using games, then we need to use professionally developed tools for healing while designing them. The psychological model is an a necessary perspective during the design of health-motivated games.

One Dish, Many Cooks

The idea of different perspectives comes into play (no pun intended) in game design when we talk about the human factors behind health-based gaming. Recent studies have found evidence that may explain the convergences as well as conceptual differences between the different experts. Obviously, game design experts are most sensitive to the mechanics of the game, but they tend to prioritize the player’s autonomy during the experience. They view the integration of gameplay and health behavior in terms of two distinct concepts. On the other hand, health experts interpret player autonomy in the context of health. They are more likely to comment on the “fun” games in contrast to the “serious” games, and they are more likely to discuss game and health concepts less in the context of integration but rather in terms of a causal relationship — game mechanics were to model health behavior. In contrast with these single-discipline groups, games for health experts view content and interaction of the while emphasizing the outcomes and objectives of both the games and health behavior. According to games for health experts, the game mechanics — its own separate entity — are responsible for producing health outcomes. These findings can be applied as conceptual tools during the design process to make sure games are made with the intent to produce desirable health objectives.

Why Do I Care? Why Should You?

I’m nearing the end of my master’s program for applied cognition and neuroscience, and I’ve been spending its duration performing cognitive tests (testing memory and attention) on volunteer participants during a cognitive training regimen. What is the cognitive training in question? Any guesses?

If you thought “games,” you thought right. And since taking a special interest in human-computer interaction theory during my undergrad years, I can’t help but speculate on the relationship between the user experience of the games that we use for cognitive training and the behavioral and neurological effects that we investigate.

Are games the future of cognitive and physical health? I can’t definitively say that for sure, but I can be pretty confident that digitizing and gamifying health is and will continue to be an important tool in a holistic approach to manage health. The studies I mentioned earlier make it evident that the technological component of health games is not only important in terms of validity but also usability. If we plan to continue implementing games into behavioral health and cognitive training as aids in metrics, management, and even enhancement (in the terms of cognitive training), we have to pay attention to the multiple factors that go into designing the games themselves. At the risk of sounding corny, I feel like the conclusion I drew from this little investigation could be summed up in a little phrase: better game design allows for better user health.

References

Digital Emotes

Using​ ​Biofeedback​ ​and​ ​Facial​ ​Tracking​ Technologies to Abstract Emotions.

In his 2018 Master’s Thesis, Aaron Tate details his project to “create​​ an artistic ​visualization ​​that ​​would ​​use both electroencephalogram ​and ​​facial ​​tracking technologies ​​as ​​input​​ devices driving creative compositions in a digital interface. ” Taking influences form data visualization, cognitive science, art, and aesthetics, Digital Emotes is an interesting look at the integration of artistic visualization and neuroimaging to produce a visual reflection of current emotional state.

Cybersecurity in the Times of Pandemic

Cyber security refers to the body of technologies, processes, and practices designed to protect networks, devices, programs, and data from attack, damage, or unauthorized access. Cybersecurity may also be referred to as information technology security. These cyberattacks are usually aimed at accessing, changing, or destroying sensitive information; extorting money from users; or interrupting normal business processes.

As we continue in our Redesign phase, we have established the idea of having a cybersecurity team. We have so much vulnerable files and data which are prone to serious attack. I was informed about the recent attack on Creative disturbance, where the hacker demanded bitcoin payment. If we had a Cyber security plan, we would have had better methods of not only handling the attack, but also preventing the lab from future attacks.

Cybersecurity is typically divided into two parts. The software, and the human behavior. The latter is the more difficult part. This plan will not only touch on software, but also the human behavior, teaching lab members what to do in order to prevent attacks, and generally safe computer and internet practices.

UTD ARTSCILAB Lab Ambassador Appointed to enable ArtSci connections Abroad

February 2020, the University of Texas at Dallas ArtSciLab appoints Jacob Hunwick as Lab Ambassador for the duration of his study abroad program in Germany. He starts at Phillips University Marburg on February the 18th and finishes on June the 12th. In addition, lab director Roger Malina appoints Jacob as an intern representative for the Leonardo Journal in Europe. 

Jacob will work to research, discover and document exemplars of art-science and well-being. Through his studies in ATEC at UT Dallas, Jacob has found a passion for technologies that prioritize the preservation and promotion of healthy habits and lifestyles. 

Through his weekly blog posts, he will report on interviews, events, and interactions with new organizations and people related to technologies that prioritize human health. 

The following is a summary of his research interests that he will pursue and write about in his weekly blog.

Research Goal for Lab Ambassador Position

Ideally, interaction designers want interfaces designed for everyday use to develop into healthy habits. Unfortunately, screen-based interfaces and modern city infrastructure trends promote sedentary habits.  

Infinitely scrolling pages and endless content tunnels enable users to over-dose on screen-time. Common use of screens for education, entertainment, and leisure time encourage people to abandon physical activity. And lastly, American city infrastructures discourage walking with a hyper focus on the automobile. 

Through my research, I seek interfaces with modern technology that improve human well-being. I seek infrastructure that empowers us to rely on our legs, not motors, to travel and navigate urban environments. I seek products that involve motion and break through the 2-dimensional touch screen barrier. I seek educational tools that encourage children to learn through active motion and participation rather than passive consumption.  

Trough the theories of embodied cognition, designers know that external objects can influence our cognitive processes. Now, the field of interaction design has realized the power that designed objects and experiences has over how we understand the world. While abroad, I will search for and document exemplars of health-conscious technologies that use the theories of embodied cognition to build healthy habits. 

To those interested in my research goal contact me via email at jmh170830@utdallas.edu. I look forward to traveling all around Europe in pursuit of my mission. 

-Jacob Hunwick 

ArtSci Abroad logo created by Jacob Hunwick.

Chess Re-Imagined

By Zura Javakhadze

         Graphic by Jacob Hunwick

On Saturday October 5th, 2019 a public performance exhibition was given at the conclusion of the month long ‘Good Moves’ exhibition dedicated to the game of chess, at The Power Station art space in Dallas, Texas. ‘Good Moves’ featured chess related artworks, that further develop the aesthetic legacy of the game, while collectively serving a worthy purpose. All works included in ‘Good Moves’ were auctioned at the close of the exhibition to endow a chess program at Vogel Alcove, a Dallas-based, non-profit organization on a mission to help young children overcome the lasting and traumatic effects of homelessness.

The performance was called “Chess Re-imagined” as was the first public exhibition of months of work examining the idea of using the game of chess tied to various multimedia configurations to display the state of the game via extended visual and auditory/musical means. Blindfolded chess game as well as all other chess related nuances were taken care by Grandmaster Elect and 3x Texas State Chess Champion Zura Javakhadze. The auditory work was based on the Data Stethoscope project of the ARTSCI Lab of UT Dallas, with the leadership of Award-winning composer Scot-Gresham Lancaster. The visual work was done by Dr. William Thibault PhD semi-independently from Auditory. Another collaborator of the project was Machine Listening and Audio Cognition expert Sharath Chandra, who was in charge of tech-aspect of the performance. 

 We are happy to report that, while a little chaotic in the context of a gallery serving alcohol and snacks in the performance area, the results of our survey and our informal inquiries have returned almost completely positive feedback and many rich insights for improvement.

Configuration 

  1. DGT electronic chess board 
  2. Software reading the chess moves on that board in real time 
  3. Those moves turned into OPEN SOUND CONTROL (OSC) messages
  4. Interpretation of those messages into: 
    1. Visual real time interactive 3d scene graph representations 
    2. Auditory Sonification that represent aspects of the game play

Results

By all our measures this event was a success. When the games were happening, the audience was engaged and as the survey demonstrated, for the most part, that the comprehension of the sonification aspects of the work were tangible and fairly well comprehended. Below is a set of materials that the performance generated that will help understand if we are able to achieve our objective analyzing this performance as an authentic instance of research. We did this in a number of ways.

Result Notes by Scot-Gresham Lancaster

This is a very positive outcome indicating that technique of using a combination of Earcon’s and Spearcon’s to reflect the chess moves was successful. This supports the notion that this approach could be transferred to other areas of data science beyond tabletop gaming which is the general area that we are categorizing this research in terms of a potential minimum viable product (MVP). There was some disappointment that we used words to express the “rank and file” (A – 7, E -3, etc.) In rehearsal we tried just sounds with much less success. So, the determination was to insure Zura’s success we would just use the timbre and Earcon melody fragments to represent the piece rank (pawn, knight, queen, etc.) and the color of the piece (black and white)

A positive result for the core part of the test, but if we get the opportunity to try this again, we need to test at a more granular level. We did not differentiate between the Earcon/Spearcon representation, the pan/pulse eval representation and the eval represented as a variable “boom chick” rhythm. There was an oversight that will be corrected next time to have a more thorough explanation of the “1st order” sonification representations that were the “pan/pulse” and “boom/chick” part of the audio experience. Roger Malina after the hectic party atmosphere of this art gallery context suggested that we arrange an invitation only “exhibition” next time. 

We did not get the opportunity to more granularly ask about distinguishing the major events of the game that were also represented with Earcons, castling, capture, check, game over (resign and tie) and checkmate. 

Additionally, and completely unexplained to the audience was that there were sonic representations of the potential future attacks. This was a very sophisticated sonic design that reflected the future state of the board and the potential new moves and their relative strengths for the side that had just moved. (Picture shown below.) Ultimately, we decided these sounds were an interesting part of the soundscape generated by the chess driven sound state machine that had been designed, but an explanation and expectation of any audience member being able to discern this sophisticated sound would not be fruitful. So, it was decided to just add it in during moments later in the evening as and an additional part of the sound texture, but unexplained.

This is a little of a more disappointing result with just a little over 20% of the respondents feeling that the overall effect of the various sonification techniques were helpful in following the game play. With more testing it is possible that the time it takes to “learn” the sounds will be able to be determined, but in this preliminary test that was not an option.

This aspect of multi modal media, or media of at least two senses is a very interesting area to pursue further. The examples using the fMRI data from our earlier research were never tested in this way, so getting these results is very telling. The intuitive conclusion is that the visuals are a good support for improving the response to the sonification, but an interesting test in a more controlled environment. The implicit future test would be a combination of just the visuals, just the sonification, and then the combination of both, to see what would get the best result.

POINTS OF FAILURE AND POTENTIAL IMPROVEMENTS

The responses of survey takers

  • It is clear that there was too much ancillary sound that was not part of the sound design that bothered some of the respondents enough to make a comment. This is a small point but worth observing. There has been a point of controversy between at least two of the collaborators. 
  • SGL felt that the playing of a non-data driven drum machine during all the breaks between games, which was a unilateral and not agreed upon part of the sound environment was an extreme distraction from our intended purpose. The other collaborators, besides SCR, agreed this seemed a distraction to the focused listening intent of this work and created a “party atmosphere” that promoted loud talking and much less attentive listening than we had desired. However, the sudden stop in the music before the actual performance resulted in the audience’s saliency and attention towards the MC taking over the microphone. This perhaps retained a critical mass of the merry audiences’ presence and attention during the actual chess performance which we all agree was a success. 
  • SCR was of the opinion that the ancillary “background music” the respondents were referring to was instances  like the one 5 minute section where an acoustic guitar improvisation was done based on the Rank and File data of a fast game play between professional players with the unexplained sounds late in the evening that were introduced representing future attack sonification.
  • This is the sort of informative result of investigation that makes our approach of performance-oriented research so unique and we would argue, richer and more compelling. 
  • A very disrespectful and glaring error was that we did not run through the talk a second time in the course of the evening and in the initial presentation SGL moved on to the first game before both Dr. Thibault and Mr. Chandra got the opportunity to fully explain their contributions to the game data representations. This was an unfortunate oversight that will be rectified if the opportunity to do this again arises. 
  • The first game between two members of the audience went on way too long. While this game was supposedly on the clock, we did not enforce the time constraints due to the inexperience of players, and it went on way longer than what was prudent. Additionally, by the end of the game it was just a back and forth. That would have been a draw in a professional tournament setting. When we revisit this, we need a mechanism for strict enforcement of time so that the games don’t go on so long. 

ETHICAL AND LEGAL CONSIDERATIONS

How did this collaboration happen?

Dr. William C. Thibault Ph.D. and Scot Gresham-Lancaster MA/MFA have been collaborating together on various pieces since 1990. When James Stallings contacted Scot to contribute to the “Good Moves” exhibit closing evening, he first thought of resurrecting the piece that he had done for the 9e2 event in Seattle in Oct. of 2016, it was a third of the presentation that was part of the DARPA funded research identified as the “Data Stethoscope” project.(Video of the full performance https://vimeo.com/356344252) This project was a demonstration of artistic approaches to using data to examine a field of FMRI data provided by neuroscientist Dr. Gagan Wig and his lab from the Center for Longevity Studies at UT Dallas.  Scot determined that this code was still working and accepted the offer to start work on the performance in July. However, this did not fully meet the criterion of moving the Data Stethoscope tests of artistic practice as research methodology. 

For the past two years Scot, Dr. Roger Malina, Sharath Chandra Ram with some interaction with various ancillary individuals have been working developing a “fuzzy taxonomy” of “Ways of Listening to Data” based on various prototype modules that both Scot and Chandra had been collaborating to develop. The intent of finding an opportunity or opportunities to develop use cases and performances to create a minimum viable product (MVP) demonstrations of this research. 

By chance, Scot reconnected with his old friend and collaborator Bill (William above) and discovered Bill had immersed himself in an in-depth private research practice into chess. Playing at a professional level and integrating some of his post-retirement research to developing chess related software to analyze the game play. 

In the ensuing conversations Bill became interested in using his own DGT chess board to create a set of a type of messages called Open Sound Control or OSC messages that would characterize each move of the chess game in real time. Working in collaboration with Scot they developed an agreed upon set of messages that represented the game play after each move. 

Once he developed this code base both he and Scot had an agreed upon set of descriptors for all the various aspects of game play, he developed a “PGN player” that sent each move from a large catalog of Chess Master games one move at a time at a random interval of 3 to 15 seconds, to emulate actual game play. A PGN is an encoded list of the algebraic notation of a chess game move by move. Each move was communicated over the internet from Bill’s studio in Castro Valley. Over the next 2 months Bill and Scot worked to expand and refine the visuals and sound separately, but in an integrated way via these OSC shared vocabulary. 

On Scot’s return trips one week a month to Texas as part of his association with the ArtSci lab, he met the Grand Master elect Zura Javakhadze to discuss the specific chess aspects of what was being developed. Also, as part of the more far reaching work of the more general Data Stethoscope project the progress of this work was shared and commented on by Dr. Roger Malina and Sharath Chandra Ram as part of their meeting regarding the ongoing research.

Besides this single face to face visit in late August, for the 2 months leading up to the performance, when Scot was back in California, there were daily email and phone connection with Roger who was introduced to the notion that Bill Thibault was working very closely with Scot creating this new work. Chandra was getting ready for the qualifying exam for his PhD and Roger and Scot agreed to lighten his load relative to this particular endeavor. This was a change from our usual configuration where Scot and Chandra worked in close daily collaboration.

The week before the performance began integration of the research modules that both Bill and Scot had been working on. Chandra was expected to have another of our earlier developed modules ready for integration from our Jupyter notebook research earlier in the year, but he had been busy with other pressing matters when we first arrived. He was able to put together a version of the “pan pulse” sonic value representation. His module, like other dynamic messages in the system was linked to the Stockfish Chess analysis engine. In this case it was a representation of the dynamic evaluation of whether White or Black was more likely to win, changing with each move. The day before the performance Chandra’s Pan/pulse eval part of the module was integrated. Additionally, Scot added a version of the “boom – chick” module. (Visual representation of Stockfish evaluation window can be seen on the right side of the image) 

Throughout the time leading into the arrival of the California part of the team, (Scot and Bill.)  Roger assigned the set of marketing and PR tasks leading up to the event, to a team of Zura Javakhadze, Linda Anderson and Jacob Hunwick. At a certain point Roger commented on the gender imbalance in the team, as we are striving to keep that balance as part of all the work associated with the ArtSci Lab. Daily input and suggestions that Scot’s wife Kathryn was contributing to the work was immense, and was acknowledged from early on. Still not a complete balance of gender but her perspective was invaluable in many of the decisions and editions as the project moved forward. Once all the collaborators were in the Dallas area, the idea was floated to recruit an “MC” for handling the flow of the program. The individuals suggested were rejected by the team that had been working on the piece. A post performance observation is that stage manager and a tighter adherence to the script would have added to the general clarity of the performance. This was a complex and multi layered collaboration that was ultimately very successful in demonstrating the efficacy of using Earcons, Spearcons and a type of 1st order sonification to represent the dynamic course of a real time chess game auditorily.

Were innovations created in the form of new intellectual property?

  • At its core this was a test to see if certain of identified sonification techniques could integrate with Dr. Thibault’s separate visual engine, much in the same way sonification for the Make or Break Unity engine were tested during the DARPA funded portion of the research. This proof was successful. 
  • We confirmed that a complex task like playing a game of chess blindfolded is possible with only auditory cues. We need to take the next step and make the cues entirely non speech earcons completely.

Was everyone’s contribution fairly credited and exhibited? 

  • Zura and Scot made an extreme blunder by not carving a time out during the performance itself for both Dr. Thibault and Sharath Chadra to describe their contributions to the crowd. This was not only bad form but it left many in the audience without clarity about the processes both visual and to a small extent auditorily that they were experiencing. Additionally, there should have been a recap announcement that credited everyone involved which was not done. That was an extreme oversight. 
  • Care was given to credit everyone involved in all the written material, including the pamphlet and poster that were generated. The announcement given to ATEC for publication included no names which was probably for the best since it was submitted the full team was still in flux.

Was everyone fairly compensated?

  • The Powerstation art space provided both the airfare and lodging for Scot and Bill 
  • An honorarium from the charity for Bill Thibault satisfied his requirements 
  • Zura and videographer Adnan Naseem were compensated by the charity fund

Success Criteria

The motivation in doing this work was driven by certain criterion which should culminate in answering these questions: 

Did the performance work as research?

  • Audience size
    • Headcount throughout the span of entire 3 hour was at about 250 people. 
  • Quiz Handout Responses
  • Number of potential investors participation 
    • 2 Interested Parties

How did it improve or add to the Data Stethoscope Project software base? 

  • From the survey results it can be asserted that future use of well thought-out musical Earcons and Spearcons can be successful.

Was the addition of artist driven research a means of coming up with innovation and insight? 

  • The positive outcome points to the fact that using a public exhibition of a well-known tabletop game that is linked to sonication is an effective way of demonstrating larger principles. More work would be needed to confirm that this specific auditory augmentation of the game of chess could be successful if distributed at a large scale.

Press reaction generated by the performance

CONCLUSION

Without a doubt this was a groundbreaking new approach to “reimagining” what tournament chess play could entail. By augmenting the sensory information created by the game play itself, we have elevated the ancient game to a new level of engagement. The results of this performance research were very positive and lead to new avenues to continue our ongoing investigation into the bringing audio user interfaces to fully distributed use across a much larger demographic than we currently see.

We can be confident that the idea of using a combination of Earcons and Spearcons was a success in guiding a blindfolded chess master through a game without moves being directly told to them and only using those more abstract sonification, once he has learned them. This supports the argument put forward in much of the writing on this topic that the idea of LET or Learning, Exploring, Testing is supported in this research performance context. Chess Master Zura became familiar with the sonic representations after about 45 minutes of a practice session. He was able to successfully win 5 total blindfold games. This points to the future use of these techniques in a variety of future contexts not only related to tabletop gaming but in other areas of research, medicine and business analytics. 

Additionally, we had enough success to conclude that we are able to contribute to the field of sonification via this performance development and using performance work as research.

Credits & Contributors

Scot Gresham Lancaster – Collaborator      Kathryn Gresham Lancaster – Project Advisor

William Thibault – Collaborator                   Thomas Riccio – Project Advisor 

Sharath Chandra – Collaborator                   Jacob Hunwick – PR Team

Gregory Ruppe – Space Facilitator               Linda Anderson – PR Team 

Roger Malina – Project Producer                  Rose Garrett – Helped designing initial flyer 

Micro ArtScience Actions (MASA)

ArtSciLab Colleagues:

I would like to draw your attention to the Recent YASMIN WWWASP discussion (the Who, what where of our artscience community of practice)- Led by Guillermo Munoz with several colleagues across the world (see at end of this post). The YASMIN list is an international list of artscience professionals and amateurs and observers.

The discussion is now open to any colleague on the Tasmin list- we look forward to your thoughts: If you aren’t on YASMIN:

I just posted this to YASMIN

“I would like to pick up on Salome Cuesta proposal for ‘micro-actions’ in our
art-science communities of practice, and Diamond Berverly’s idea of
a ‘continous conundrum (see below).

After 30 years as Executive Editor of Leonardo, for the first time I am
now deeply engaged as a practitioner in art science collaborations. I
like to joke that astrophysics was so easy ! We all agreed on the success criteria and used the same concepts, methods and terminology. Plus we were very well funded in comparison with artscience research.

Thanks to Guillermo Munoz and colleagues at the University of Valencia I received a PhD in Art, so I am now an ArtScience ‘postdoc’ with a PhD in astrophysics and a PhD in art !

As James Leach pointed out in his Leonardo article (Extending Contexts, Making Possibilities: An Introduction to Evaluating the Projects – James Leach) many many art science collaborations ‘fail’ in achieving their original objectives.

In our ArtSciLab we have been arguing that there is a very big step between inter/multi disciplinary collaborations and ‘trans’ disciplinary collaborations which bridge very very different disciplines with very very different personal and collective success criteria, and very very different often contradictory methods, concepts , terminologies-

These are in Diamond Berverly’s term ‘conundral;- see Merriam Webster

“The exact origin of conundrum isn’t known with certainty. What is known is that the word has been in use since the early 1600s, and that it had various spellings, such as conimbrum, quonundrum, conuncrum, and quadundrum, before the current spelling was finally established sometime in the mid-17th century. One theory of origin suggests that the word was coined as a parody of Latin by students at Oxford University, where it appears to have enjoyed particular popularity in its “word play” or “pun” sense. While the prevalent sense in this century is that of the seemingly unanswerable question or problem, frequently applied to heady dilemmas involving ethics, sociology, or economics, the word is sometimes so loosely applied to anything enigmatic as to be synonymous with puzzle or mystery.”

One of the strategies we have been trying is “micro-projects” in Cuesta’s terminology- we define them as short duration ( weeks), projects which require no money or cash, only gift exchange of time and access to facilities, expertise etc ( the gift exchange vocabulary comes from James Leach )

In the ArtSciLab we are approached continuously by ‘stem’ professionals who want to collaborate. Interestingly these range from physical sciences and engineering to the brain and Behavioural sciences to business and social sciences…

Often the stem professionals who reach out to us, “they” have antiquated ideas of what making art involves or results in today. Often they view art as less primary in the ‘tree of knowledge” (i.e. in a branch not in the trunk).

Often they have very different methods , concepts, terminology (we are trying to develop transdisciplinary apprenticeships as part of the approach). Mauricio Mejia at ASU, Alex Topete, colleagues, and I are currently submitting a workshop proposal on applying translation studies methods to transdisciplinary collaborations.

If a micro-project doesn’t succeed, its likely that a significant project won’t in our experience.

I am delighted that a student research in our ArtSciLab , Diamond Beverly, has proposed:
“how do we go about including diverse voices and fostering heterogeneous approaches instead unconsciously excluding people from the conversation and thus creating a continuous conundrum.” which highlights the implicit bias of the artscience community towards people in academia with its ethnic, gender, socio-economic and other implicit social biases.

Maybe the non -academic and younger members of Yasmin should weigh in, just as from Diamond Berverly:
I was very intrigued with your post last week when you emphasized
educational spaces and work shop methodologies. I would like to know
how you define micro-actions. I also find such maker spaces and
hackathons a good step into the future of collaborative educational
space. A question that persists however is how these spaces find their
audience?

And by this I mean how do we go about including diverse voices and fostering heterogeneous approaches instead unconsciously excluding people from the conversation and thus creating a continuous conundrum.

So all Latin students on Yasmin go at it :
“One theory of origin of ‘conundrum” suggests that the word was coined as a parody of Latin by students at Oxford University, where it appears to have enjoyed particular popularity in its “word play” or “pun” sense. While the prevalent sense in this century is that of the seemingly unanswerable question or problem, frequently applied to heady dilemmas involving ethics, sociology, or economics, the word is sometimes so loosely applied to anything enigmatic as to be synonymous with puzzle or mystery.”

Or maybe living in a continuous conundrum is a desirable state ? Certainly it is a state that drives our ArtScience work and is generative of desired outcomes.

Roger Malina

P.S These are the invited discussants:

Jadwiga Charzynska: I am the director of LAZNIA CCA in Gdansk since 2004 and in our program, one of the most important points is the Art + Science Meeting project, which we organize regularly from 2011. It’s really unique program in Poland presented Art / Sci projects.
Links: http://www.laznia.pl/instytucja/

Joao Silveira: Brazilian entrepreneur, choreographer and pharmacist. Was a Harvard Fellow (Le Laboratoire/ 2017-2019) and currently is a research fellow at the ArtSciLab – UT Dallas, and CienciArte Lab – Fiocruz.
Jing Chen: Associate professor of Arts and associate director of Art and Cultural Innovation and Creativity Lab of Nanjing University.
Gustavo Ariel Schwartz.: Physicist and writer. Scientist of the Spanish National Research Council (CSIC) at the Materials Physics Center. Founder director of the Mestizajes Program, at the Donostia International Physics Center, whose purpose is to explore and cross the borders among Art, Science, Literature and Humanities.

Esports Cyberathlete Development (ECD): An Annotated Bibliography

The Initiative to Enhance the Esports Player Cognitive Performance & Wellbeing.

This Esports Cyberathlete Development (ECD) project launched by the University of Texas at Dallas’ ArtSciLab aims to contribute to design one of the first new sports of the 21st century.

Our efforts aim to:
(1) Promote positive values obtained from participating on an Esports team;
a. Investigate the enhanced cognitive abilities through the strategically-complex gameplay of Esports
b. Investigate the positive social developments that occur in and out of the Esports environment
(2) Focus on relieving the stigmatized universe of Esports concerning gender stereotypes and effects on intellectual performance;
a. Our research aims to increase overall inclusivity among various demographics and how its benefits are not discriminatory towards any gender
(3) Help to integrate the skills learned by the Esports players into their lives during and after their Esports careers;
a. Analysis of cognitive skills acquired by gamers during their gameplay, how such skills can generally transfer to real-world application, and to increase their executive longevity.