Franklin Osuagwu transitions out of role of Lab Coordinator

Franklin Osuagwu’s timeline as Lab Coordinator at the ArtSciLab.

Franklin Osuagwu started working at the ArtSciLab in June 2019. After the exit of Kyle, he came in as the new Lab Coordinator. While working at the lab, his major duties were organizing the MOWG meetings, organizing the weekly watering hole events, and finally creating a cybersecurity plan for the Lab. As he steps ahead in his career path moving forward, he conducts a final interview with Alex Topete on his experience and next moves.

What was his first Introduction to the ArtSciLab?

Franklin initially had no prior knowledge about the Lab. Being an electrical engineering major, he had no information about activities going on in ATEC, least of all, the Watering Hole. He found about the lab through the lab coordinator job posting on Handshake last summer.

How did Franklin rate his experience at the ArtSciLab?

Franklin loved his time at the ArtSciLab. He always found a chance to speak about how Roger and the rest of his coworkers were. He was able to make new valuable and professional connections. One thing he mentioned he loved mostly the lab was that it also served as his new “chilling spot” in between his classes.

Is there any new skill that Franklin picked up through his experience?

Franklin indicated that his management skills were on an all time high. His constant interaction with people in the lab helped him know how to manage and deal with people in an ideal work environment.

Difficulties while working at the Lab?

Franklin mentioned that one of his major issues was communication. In terms of people responding to emails on timely manner. He further mentioned that he would have loved a better attendance of lab events by its members, more especially the Watering Hole held weekly.

Where is he heading next?

Franklin is currently working as an IT intern at Epsilon. His internship is expected to run from January till December when he graduates. He will still maintain relationships with the Lab, serving as a Lab ambassador.

His final note his former teammates and coworkers are “Be sure to catch me at future watering holes.”

UTD ARTSCILAB Lab Ambassador Appointed to enable ArtSci connections Abroad

February 2020, the University of Texas at Dallas ArtSciLab appoints Jacob Hunwick as Lab Ambassador for the duration of his study abroad program in Germany. He starts at Phillips University Marburg on February the 18th and finishes on June the 12th. In addition, lab director Roger Malina appoints Jacob as an intern representative for the Leonardo Journal in Europe. 

Jacob will work to research, discover and document exemplars of art-science and well-being. Through his studies in ATEC at UT Dallas, Jacob has found a passion for technologies that prioritize the preservation and promotion of healthy habits and lifestyles. 

Through his weekly blog posts, he will report on interviews, events, and interactions with new organizations and people related to technologies that prioritize human health. 

The following is a summary of his research interests that he will pursue and write about in his weekly blog.

Research Goal for Lab Ambassador Position

Ideally, interaction designers want interfaces designed for everyday use to develop into healthy habits. Unfortunately, screen-based interfaces and modern city infrastructure trends promote sedentary habits.  

Infinitely scrolling pages and endless content tunnels enable users to over-dose on screen-time. Common use of screens for education, entertainment, and leisure time encourage people to abandon physical activity. And lastly, American city infrastructures discourage walking with a hyper focus on the automobile. 

Through my research, I seek interfaces with modern technology that improve human well-being. I seek infrastructure that empowers us to rely on our legs, not motors, to travel and navigate urban environments. I seek products that involve motion and break through the 2-dimensional touch screen barrier. I seek educational tools that encourage children to learn through active motion and participation rather than passive consumption.  

Trough the theories of embodied cognition, designers know that external objects can influence our cognitive processes. Now, the field of interaction design has realized the power that designed objects and experiences has over how we understand the world. While abroad, I will search for and document exemplars of health-conscious technologies that use the theories of embodied cognition to build healthy habits. 

To those interested in my research goal contact me via email at jmh170830@utdallas.edu. I look forward to traveling all around Europe in pursuit of my mission. 

-Jacob Hunwick 

ArtSci Abroad logo created by Jacob Hunwick.

A Spoken Word on Life and Death

Dr. James L. Carter, geoscientist, and associate professor emeritus at the University of Texas at Dallas, passed away on September 21, 2019, in his home at the age of 82. To honor James Carter, ArtSciLab member Ayen Deng has written and performed a spoken word poem in memory of the way he inspired all those who attended his last lecture on September 20, 2019. The slides in this video are from his lecture to accompany the performance.


A Review of Virtual Menageries: Animals as Mediators in Network Cultures

by Michael Warner

Virtual Menageries puts us back in time amongst the collective elites. Berland forms a cohesive genealogy of the “menagerie” to encourage, challenge, and deconstruct our modern perception of non-human animals and their relationship to human meaning and existence. Virtual Menageries looks through the lens of mediation to draw affective and emotional weight to animals as symbolic messengers in the digital era. From the giraffe, to the beaver, virtual art, digital communications, cats, birds and music Berland maps out how animals have become not only the mediators that bridge worlds together for good, but also as the trafficked subjects of terror: tools made to control, methods for silencing, opportunities to proliferate a message, and catalysts in profiteering.
 
The book begins with a question that led to a series of other questions: “Why are there so many cats on the internet?” (Pg. 1) Which then led to thoughts about the roles of animals as symbols and figures in contexts. “How do animals help manage our perception of the Anthropocene? How do they disrupt our own relationship with digital technologies if they are so abundantly apart of them? If they are mediating in new ways, what content are they mediating, and in what context?” (Pg. 1) These are all questions that threw the book and Berland’s project into fruition.    

To read the full review:

ArtSciLab Talkshow: A new series on The Bold Roast

Produced and created by Maisha Razzaque, a new series has launched on The Bold Roast: Student Conversations channel on Creative Disturbance.

The ArtSciLab is home to collaborations between artists and scientists who investigate topics such as experimental publishing, data sonification, data visualization, and the hybridization of art and science. This series is an audio experience that allows the spotlight to fall on its members as they talk to us about their careers, contributions, and passions.

Click here to listen to the first episode of The ArtSciLab Talk Show.

The Voices of Science

Before things were written they were spoken. The Spoken Word has a rich historical basis, especially amongst traditional African societies where culture and knowledge was passed down in the form of riddles, proverbs, stories, poetry, music, and design. Today, spoken word remains a fundamental form of communication, though its limits in academia are rarely challenged. Spoken word poetry is a tool to communicate social issues. Today, it is increasingly popular among the youth with so-called ‘poetry slams’ happening all around the world. Spoken word is appealing as it is impactful and lawless. There are no literary restrictions that define what it is. Instead, it takes a more performative approach, aiming to reach — even interact with — its audience; it is centered on involvement and exchange.This is what makes spoken word, as a type of poetry, powerful: It surpasses communication and creates a participatory audience. Contrastly, scientific phenomena — especially with increasing reliance on technological tools — long ago left the realm of our physical experiences. Consequently, there expands a chasm in intellectual exchange across science and other disciplines that calls for the expertise of a poet. The poet’s role will be to create innovative, metaphorical models in words and to express the often abstract and intangible phenomena in science. The very nomenclature of science, which is often times misleading, could benefit greatly from the collaboration of a poet.

It is the ability of a word to transmit meaning from one consciousness to the other that has significance and power. There is a biblical story of people of one language, building a tower with the intention to reach God. Eventually God decides to confuse them by mixing their languages — thus, the place was given the name Babel, meaning a confusion of voices in Hebrew. The sudden shift in communications, one might imagine, would lead to the development of diverse cultures and ideas. Therefore, metaphorically speaking, the growth of the tower was no longer able to be focused on only one dimension.

To a large extent, the language of science is mathematics but supplemented by words, diagrams, or images, each of which acts as a model to communicate reality. Going deeper into the study of science, particularly physics, it becomes impossible to deeply understand, let alone explain, phenomena without mathematics. One can see mathematics, the main language of science, taking a tower-like trajectory; It becomes increasingly complex and eventually, too high for unspecialized populations to reach and interact with. And when things cease to have the capacity to be understood and influenced; then, they lose their power to progress and diverge through otherwise diverse minds.

The word ‘Science’ itself carries heavy cultural connotations. Science could be seen as a dreaded school subject, a subject that is distant for people unexposed to its exciting study. How the scientist sees him/herself depends on their level of experience as a scientist. Personally, science has evolved from a de facto puzzle of a classroom study to one where there is a lot of structured seeking with a lot of room for speculation, interpretation, mistakes, evolution, and a lot of meticulous tedious work and creative planning.

Ideas of scientism stating that science is a closed box, superior to all other modes of intelligence, not only limit but harm our society.

Science affects everyone and exists in all of creation. It is understood in one way by scientists another way by artists, poets, spiritualists and other disciplines. All these distinctions are relevant for practical purposes. They are not laws. Our strength and integrity as a society will be found in open exchange between science and the other disciplines. Such permeabilities are what will allow us progress in multiple degrees of freedom, adding wealth to science studies and how we as a diverse persons view and interact with it.

One of the entry points in which such exchange can occur is our reliance on models to understand and discover new things. The very model for how learning takes place includes formation of new networks of knowledge upon already existing ones. Our minds work like an intricate web making connections in order to understand and develop ideas. In her book Models and Analogies in Science, Mary Hess makes reference to positive, negative, and neutral analogies. Negative analogies being those that we know are unable to fit into a description, positive being those that agree, and neutral being those that are unknown and have the potential to be investigated. This is where spoken word poetry comes in. Poetry would excel at making connections between science principles and unexpected elements of life, juxtaposing vivid imagery which enlivens striking metaphors and narratives — engaging the scientist, science, and everyday life.

For example the verse below:
“Our consciousness , so close, yet so distant, allows us to travel at the speed of light when we fall in love ;(that’s about a 24 times a year for me- twice a month before and after ovulation) But like two ends of the same string, we sink to normality in the greyness and redness of stuff. Though we are made of things that are the substance of light , we can only pulse in inconsistency”
This describes how time dilation, that occurs in general relativity, is the same kind that is experienced by humans when they are focused or feeling intense emotions such as pain or love. One can model traveling at the speed of light to be analogous to being deeply focused or in intense enjoyment where the actual time is moving much faster than the time internally experienced. It also touches on the wave particle duality, and the relationship between physiology and personality.
Spoken word could lead to a plethora of analogies with the potential to be sorted and investigated. Neutral analogy is just one of the pathways that could lead to research investigation, thereby spoken word poetry is a prime example of art as a research method. It can clearly be used in learning. It’s not uncommon for fantastical scenarios such as: “ What if you found yourself in space holding a….” to be used in a classroom question, but it is often not taken further. Though metaphors might shift from their origin, they always find their way back in some form. Vital is the kind of imagery and metaphorical tension existent in engaging spoken word narratives that trigger the mind’s imagination in ways that information in itself could never dream of.

“Imagination is more important than knowledge. Knowledge is limited. Imagination encircles the world,” (Albert Einstein).

Spoken word most importantly holds the power to open room for discourse between unexpected combinations of people.

There seems to me, a great potential to develop scientific spoken word exchanges for the stage, research, learning, creating art, and cultural revolutions.

References

“What is Science” by Sundar Sarukkai
“A review of African Oral traditions and literature”: by Harold Schlub
“Ways of Seeing” by John Berger
“Making science intimate” by Roger Malina
“Science et cetera et cetera for poets et cetera” by John A Moore
“Genesis” Judaic Bible

ArtSciLab Promotes Three Researchers to Leadership Positions

The ArtSciLab has named Dr. Kathryn Evans, Associate Professor of Instruction in the School of Arts & Humanities of UT Dallas, as Co-Director of Education Initiatives, while also promoting Dr. Yvan Tina and Alex Garcia Topete, two of its longest serving ATEC researchers, to the positions of Deputy Co-Directors of the lab. The leadership promotions come as part of a major re-design of the systems and processes of the ArtSciLab as it adapts to the challenges of managing research, strategy, organizational culture, and operations among a growing group of student lab members and external collaborators. 

Dr. Evans, as Co-Director of Education Initiatives of the lab, will oversee the ArtSciLab’s projects that merge diverse disciplines in learning and teaching settings, particularly those involving the growing trend of turning education from “STEM” to “STEAM” education through the arts. Dr. Evans brings to the position not only her years of experience as an educator, but also her many talents as an academic researcher, singer, conductor, director, and producer. The ArtSciLab’s current education initiatives include CDASH (Curriculum Development in the Arts, Sciences and Humanities), ToTTS (Tale of Two Thinking Systems), and Arts-Based Learning for Business (ABLB)–all which were originated by Dr. Evans as a lab researcher. 

Dr. Tina will now serve as Deputy Co-Director of Performative Research of the lab. Dr. Tina’s expertise is in performance studies and he will oversee the research projects related to new forms of representation and curation of the arts, with an emphasis on scenic practices involving artificial intelligence and biotechnologies. His role will be to stimulate and strengthen the artistic output of the ArtSciLab. As a performative researcher, he is a founding member and manager of the multilingual and international podcast platform Creative Disturbance, which requires the coordination of a team composed of UX/UI designers, sound designers, and web developers.  Dr. Tina is also a member of the Laboratoire d’Études en Sciences des Arts (LESA) of Aix-Marseille University and the artistic director of the Virtual Africa initiative of Leonardo/Olats. He lectures in a number of different venues and conferences and will participate this year to the Next Einstein Forum. 

Mr. Garcia Topete will function as Deputy Co-Director of Transdisciplinary Strategy and Collaboration. His main duties will include overseeing the Experimental Publishing and Curation (ExPuCu) efforts of the lab, implementing systems and workflows with the aim of fostering creativity, innovation, and collaboration, and designing transdisciplinary strategies for teamwork and successful knowledge management. His almost two decades of experience in the creative industries as an international filmmaker and film festival organizer, as well as more than a decade in the entrepreneurial and non-profit world, make Mr. Garcia Topete particularly equipped with a multifaceted understanding and transnational perspective of knowledge exchange and social impact to deal with the current and future challenges of the ArtSciLab.  

“As the African proverb says: if you want to go fast, go alone; if you want to go far, go together; and we’re definitely trying to go far in our lab,” said Founding Executive Co-Director Dr. Roger Malina about the reasoning for the lab reorganization. With these new leadership positions and overall redesign, the ArtSciLab seeks to improve not only its operations and management, but also provide a better learning environment and culture in which its members and collaborators share knowledge more effectively, mentor each other in their areas of expertise with more ease, and develop further both professionally and personally through their lab participation. Ultimately, the ArtSciLab continues to pioneer in transdisciplinary issues—this time looking to boost the careers and development of everyone involved. 

Chess Re-Imagined

By Zura Javakhadze

         Graphic by Jacob Hunwick

On Saturday October 5th, 2019 a public performance exhibition was given at the conclusion of the month long ‘Good Moves’ exhibition dedicated to the game of chess, at The Power Station art space in Dallas, Texas. ‘Good Moves’ featured chess related artworks, that further develop the aesthetic legacy of the game, while collectively serving a worthy purpose. All works included in ‘Good Moves’ were auctioned at the close of the exhibition to endow a chess program at Vogel Alcove, a Dallas-based, non-profit organization on a mission to help young children overcome the lasting and traumatic effects of homelessness.

The performance was called “Chess Re-imagined” as was the first public exhibition of months of work examining the idea of using the game of chess tied to various multimedia configurations to display the state of the game via extended visual and auditory/musical means. Blindfolded chess game as well as all other chess related nuances were taken care by Grandmaster Elect and 3x Texas State Chess Champion Zura Javakhadze. The auditory work was based on the Data Stethoscope project of the ARTSCI Lab of UT Dallas, with the leadership of Award-winning composer Scot-Gresham Lancaster. The visual work was done by Dr. William Thibault PhD semi-independently from Auditory. Another collaborator of the project was Machine Listening and Audio Cognition expert Sharath Chandra, who was in charge of tech-aspect of the performance. 

 We are happy to report that, while a little chaotic in the context of a gallery serving alcohol and snacks in the performance area, the results of our survey and our informal inquiries have returned almost completely positive feedback and many rich insights for improvement.

Configuration 

  1. DGT electronic chess board 
  2. Software reading the chess moves on that board in real time 
  3. Those moves turned into OPEN SOUND CONTROL (OSC) messages
  4. Interpretation of those messages into: 
    1. Visual real time interactive 3d scene graph representations 
    2. Auditory Sonification that represent aspects of the game play

Results

By all our measures this event was a success. When the games were happening, the audience was engaged and as the survey demonstrated, for the most part, that the comprehension of the sonification aspects of the work were tangible and fairly well comprehended. Below is a set of materials that the performance generated that will help understand if we are able to achieve our objective analyzing this performance as an authentic instance of research. We did this in a number of ways.

Result Notes by Scot-Gresham Lancaster

This is a very positive outcome indicating that technique of using a combination of Earcon’s and Spearcon’s to reflect the chess moves was successful. This supports the notion that this approach could be transferred to other areas of data science beyond tabletop gaming which is the general area that we are categorizing this research in terms of a potential minimum viable product (MVP). There was some disappointment that we used words to express the “rank and file” (A – 7, E -3, etc.) In rehearsal we tried just sounds with much less success. So, the determination was to insure Zura’s success we would just use the timbre and Earcon melody fragments to represent the piece rank (pawn, knight, queen, etc.) and the color of the piece (black and white)

A positive result for the core part of the test, but if we get the opportunity to try this again, we need to test at a more granular level. We did not differentiate between the Earcon/Spearcon representation, the pan/pulse eval representation and the eval represented as a variable “boom chick” rhythm. There was an oversight that will be corrected next time to have a more thorough explanation of the “1st order” sonification representations that were the “pan/pulse” and “boom/chick” part of the audio experience. Roger Malina after the hectic party atmosphere of this art gallery context suggested that we arrange an invitation only “exhibition” next time. 

We did not get the opportunity to more granularly ask about distinguishing the major events of the game that were also represented with Earcons, castling, capture, check, game over (resign and tie) and checkmate. 

Additionally, and completely unexplained to the audience was that there were sonic representations of the potential future attacks. This was a very sophisticated sonic design that reflected the future state of the board and the potential new moves and their relative strengths for the side that had just moved. (Picture shown below.) Ultimately, we decided these sounds were an interesting part of the soundscape generated by the chess driven sound state machine that had been designed, but an explanation and expectation of any audience member being able to discern this sophisticated sound would not be fruitful. So, it was decided to just add it in during moments later in the evening as and an additional part of the sound texture, but unexplained.

This is a little of a more disappointing result with just a little over 20% of the respondents feeling that the overall effect of the various sonification techniques were helpful in following the game play. With more testing it is possible that the time it takes to “learn” the sounds will be able to be determined, but in this preliminary test that was not an option.

This aspect of multi modal media, or media of at least two senses is a very interesting area to pursue further. The examples using the fMRI data from our earlier research were never tested in this way, so getting these results is very telling. The intuitive conclusion is that the visuals are a good support for improving the response to the sonification, but an interesting test in a more controlled environment. The implicit future test would be a combination of just the visuals, just the sonification, and then the combination of both, to see what would get the best result.

POINTS OF FAILURE AND POTENTIAL IMPROVEMENTS

The responses of survey takers

  • It is clear that there was too much ancillary sound that was not part of the sound design that bothered some of the respondents enough to make a comment. This is a small point but worth observing. There has been a point of controversy between at least two of the collaborators. 
  • SGL felt that the playing of a non-data driven drum machine during all the breaks between games, which was a unilateral and not agreed upon part of the sound environment was an extreme distraction from our intended purpose. The other collaborators, besides SCR, agreed this seemed a distraction to the focused listening intent of this work and created a “party atmosphere” that promoted loud talking and much less attentive listening than we had desired. However, the sudden stop in the music before the actual performance resulted in the audience’s saliency and attention towards the MC taking over the microphone. This perhaps retained a critical mass of the merry audiences’ presence and attention during the actual chess performance which we all agree was a success. 
  • SCR was of the opinion that the ancillary “background music” the respondents were referring to was instances  like the one 5 minute section where an acoustic guitar improvisation was done based on the Rank and File data of a fast game play between professional players with the unexplained sounds late in the evening that were introduced representing future attack sonification.
  • This is the sort of informative result of investigation that makes our approach of performance-oriented research so unique and we would argue, richer and more compelling. 
  • A very disrespectful and glaring error was that we did not run through the talk a second time in the course of the evening and in the initial presentation SGL moved on to the first game before both Dr. Thibault and Mr. Chandra got the opportunity to fully explain their contributions to the game data representations. This was an unfortunate oversight that will be rectified if the opportunity to do this again arises. 
  • The first game between two members of the audience went on way too long. While this game was supposedly on the clock, we did not enforce the time constraints due to the inexperience of players, and it went on way longer than what was prudent. Additionally, by the end of the game it was just a back and forth. That would have been a draw in a professional tournament setting. When we revisit this, we need a mechanism for strict enforcement of time so that the games don’t go on so long. 

ETHICAL AND LEGAL CONSIDERATIONS

How did this collaboration happen?

Dr. William C. Thibault Ph.D. and Scot Gresham-Lancaster MA/MFA have been collaborating together on various pieces since 1990. When James Stallings contacted Scot to contribute to the “Good Moves” exhibit closing evening, he first thought of resurrecting the piece that he had done for the 9e2 event in Seattle in Oct. of 2016, it was a third of the presentation that was part of the DARPA funded research identified as the “Data Stethoscope” project.(Video of the full performance https://vimeo.com/356344252) This project was a demonstration of artistic approaches to using data to examine a field of FMRI data provided by neuroscientist Dr. Gagan Wig and his lab from the Center for Longevity Studies at UT Dallas.  Scot determined that this code was still working and accepted the offer to start work on the performance in July. However, this did not fully meet the criterion of moving the Data Stethoscope tests of artistic practice as research methodology. 

For the past two years Scot, Dr. Roger Malina, Sharath Chandra Ram with some interaction with various ancillary individuals have been working developing a “fuzzy taxonomy” of “Ways of Listening to Data” based on various prototype modules that both Scot and Chandra had been collaborating to develop. The intent of finding an opportunity or opportunities to develop use cases and performances to create a minimum viable product (MVP) demonstrations of this research. 

By chance, Scot reconnected with his old friend and collaborator Bill (William above) and discovered Bill had immersed himself in an in-depth private research practice into chess. Playing at a professional level and integrating some of his post-retirement research to developing chess related software to analyze the game play. 

In the ensuing conversations Bill became interested in using his own DGT chess board to create a set of a type of messages called Open Sound Control or OSC messages that would characterize each move of the chess game in real time. Working in collaboration with Scot they developed an agreed upon set of messages that represented the game play after each move. 

Once he developed this code base both he and Scot had an agreed upon set of descriptors for all the various aspects of game play, he developed a “PGN player” that sent each move from a large catalog of Chess Master games one move at a time at a random interval of 3 to 15 seconds, to emulate actual game play. A PGN is an encoded list of the algebraic notation of a chess game move by move. Each move was communicated over the internet from Bill’s studio in Castro Valley. Over the next 2 months Bill and Scot worked to expand and refine the visuals and sound separately, but in an integrated way via these OSC shared vocabulary. 

On Scot’s return trips one week a month to Texas as part of his association with the ArtSci lab, he met the Grand Master elect Zura Javakhadze to discuss the specific chess aspects of what was being developed. Also, as part of the more far reaching work of the more general Data Stethoscope project the progress of this work was shared and commented on by Dr. Roger Malina and Sharath Chandra Ram as part of their meeting regarding the ongoing research.

Besides this single face to face visit in late August, for the 2 months leading up to the performance, when Scot was back in California, there were daily email and phone connection with Roger who was introduced to the notion that Bill Thibault was working very closely with Scot creating this new work. Chandra was getting ready for the qualifying exam for his PhD and Roger and Scot agreed to lighten his load relative to this particular endeavor. This was a change from our usual configuration where Scot and Chandra worked in close daily collaboration.

The week before the performance began integration of the research modules that both Bill and Scot had been working on. Chandra was expected to have another of our earlier developed modules ready for integration from our Jupyter notebook research earlier in the year, but he had been busy with other pressing matters when we first arrived. He was able to put together a version of the “pan pulse” sonic value representation. His module, like other dynamic messages in the system was linked to the Stockfish Chess analysis engine. In this case it was a representation of the dynamic evaluation of whether White or Black was more likely to win, changing with each move. The day before the performance Chandra’s Pan/pulse eval part of the module was integrated. Additionally, Scot added a version of the “boom – chick” module. (Visual representation of Stockfish evaluation window can be seen on the right side of the image) 

Throughout the time leading into the arrival of the California part of the team, (Scot and Bill.)  Roger assigned the set of marketing and PR tasks leading up to the event, to a team of Zura Javakhadze, Linda Anderson and Jacob Hunwick. At a certain point Roger commented on the gender imbalance in the team, as we are striving to keep that balance as part of all the work associated with the ArtSci Lab. Daily input and suggestions that Scot’s wife Kathryn was contributing to the work was immense, and was acknowledged from early on. Still not a complete balance of gender but her perspective was invaluable in many of the decisions and editions as the project moved forward. Once all the collaborators were in the Dallas area, the idea was floated to recruit an “MC” for handling the flow of the program. The individuals suggested were rejected by the team that had been working on the piece. A post performance observation is that stage manager and a tighter adherence to the script would have added to the general clarity of the performance. This was a complex and multi layered collaboration that was ultimately very successful in demonstrating the efficacy of using Earcons, Spearcons and a type of 1st order sonification to represent the dynamic course of a real time chess game auditorily.

Were innovations created in the form of new intellectual property?

  • At its core this was a test to see if certain of identified sonification techniques could integrate with Dr. Thibault’s separate visual engine, much in the same way sonification for the Make or Break Unity engine were tested during the DARPA funded portion of the research. This proof was successful. 
  • We confirmed that a complex task like playing a game of chess blindfolded is possible with only auditory cues. We need to take the next step and make the cues entirely non speech earcons completely.

Was everyone’s contribution fairly credited and exhibited? 

  • Zura and Scot made an extreme blunder by not carving a time out during the performance itself for both Dr. Thibault and Sharath Chadra to describe their contributions to the crowd. This was not only bad form but it left many in the audience without clarity about the processes both visual and to a small extent auditorily that they were experiencing. Additionally, there should have been a recap announcement that credited everyone involved which was not done. That was an extreme oversight. 
  • Care was given to credit everyone involved in all the written material, including the pamphlet and poster that were generated. The announcement given to ATEC for publication included no names which was probably for the best since it was submitted the full team was still in flux.

Was everyone fairly compensated?

  • The Powerstation art space provided both the airfare and lodging for Scot and Bill 
  • An honorarium from the charity for Bill Thibault satisfied his requirements 
  • Zura and videographer Adnan Naseem were compensated by the charity fund

Success Criteria

The motivation in doing this work was driven by certain criterion which should culminate in answering these questions: 

Did the performance work as research?

  • Audience size
    • Headcount throughout the span of entire 3 hour was at about 250 people. 
  • Quiz Handout Responses
  • Number of potential investors participation 
    • 2 Interested Parties

How did it improve or add to the Data Stethoscope Project software base? 

  • From the survey results it can be asserted that future use of well thought-out musical Earcons and Spearcons can be successful.

Was the addition of artist driven research a means of coming up with innovation and insight? 

  • The positive outcome points to the fact that using a public exhibition of a well-known tabletop game that is linked to sonication is an effective way of demonstrating larger principles. More work would be needed to confirm that this specific auditory augmentation of the game of chess could be successful if distributed at a large scale.

Press reaction generated by the performance

CONCLUSION

Without a doubt this was a groundbreaking new approach to “reimagining” what tournament chess play could entail. By augmenting the sensory information created by the game play itself, we have elevated the ancient game to a new level of engagement. The results of this performance research were very positive and lead to new avenues to continue our ongoing investigation into the bringing audio user interfaces to fully distributed use across a much larger demographic than we currently see.

We can be confident that the idea of using a combination of Earcons and Spearcons was a success in guiding a blindfolded chess master through a game without moves being directly told to them and only using those more abstract sonification, once he has learned them. This supports the argument put forward in much of the writing on this topic that the idea of LET or Learning, Exploring, Testing is supported in this research performance context. Chess Master Zura became familiar with the sonic representations after about 45 minutes of a practice session. He was able to successfully win 5 total blindfold games. This points to the future use of these techniques in a variety of future contexts not only related to tabletop gaming but in other areas of research, medicine and business analytics. 

Additionally, we had enough success to conclude that we are able to contribute to the field of sonification via this performance development and using performance work as research.

Credits & Contributors

Scot Gresham Lancaster – Collaborator      Kathryn Gresham Lancaster – Project Advisor

William Thibault – Collaborator                   Thomas Riccio – Project Advisor 

Sharath Chandra – Collaborator                   Jacob Hunwick – PR Team

Gregory Ruppe – Space Facilitator               Linda Anderson – PR Team 

Roger Malina – Project Producer                  Rose Garrett – Helped designing initial flyer 

Do you want a “Data Sonification Player” for Astrophysical Data or your own data?

Colleagues:

Under the leadership of ASL member Scot Gresham Lancaster, Sharath Chandra Ram and I have just submitted a proposal to the US National Science foundation:

AstroSounds: Developing spectrally based multivariate time series data sonification player for Astrophysics

Overview:

We are going to use auditory cognition techniques of speech perception for the listening that astrophysicist would be asked to do with a spectrally based multivariate time series data player.

LISTENING TO DATA (Sonification) is a new tool needed for analyzing the ever growing data cloud. Our work is perfecting the many ways to make better decisions by not just visualizing, but also LISTENING TO DATAs This has lead us to helping others in many fields, neuroscience, business,
general physics etc. but now we are returning our attention back to astrophysics and the advantages that the integration of audio brings to the data science interfaces that are such an integral part of astrophysical research.

We have developed a new approach to classifying the techniques of data to audio conversion and the specific types of applications that can be offered using the advanced techniques available to top digital audio experts to realize those conversions. Collaborating with our colleagues at UC Berkeley, Stanford and other institutions, we have been sharing our ideas and approaches to the sonic realizations that highlight features blurred or misrepresented in visualizations. We have exposed artifacts occluded within only imaged representations of data. Also we can simultaneously represent many other dimensions not visually represented in the complex time varying graphs. Data visualization tools have successfully been augmented to allow more rapid and accurate decision making in the context of ever growing data sets.

We continue to create training modules to learn to listen, explore and pass testing of sonifed data to interpret data as information via ongoing cognitive testing. Working from our existing, tested and published “fuzzy” taxonomy of classifications and techniques we will use 2nd order sonification that remaps information systematically via spectral analysis and auditory spectrum resynthesis of time series data. Additionally, we will use our machine listening techniques to find spectral similarities in the
overwhelmingly large amounts of available data in the existing open archives. Using new techniques of binaural audio augmented reality, we will allow researchers to move through the soundscape created by time varying multidimensional data sets. This is based on our existing and cognitively tested earlier work in this area.

Broader Impacts:

The adoption of sound in the market of Smart Speakers (Alexa, Google Home, etc.) and Audio AR (Bose, Onset, etc. ) have shown the growing more dominant place for audio oriented design. For the first time we and leading with astrophysics, we will move to scale across all areas, that are becoming dependent on data science techniques and outcomes. The perception of multidimensional auditory information will lead to new discoveries that would not be made by visual means alone.