In order to bring the broad application of technology to Huatung, on March 12, Professor Alvin Su, along with his team and partners held a virtual concert at Junyi School of Innovation. It was a symposium of music and technology.
Music X Computer X AI
Traditionally one may think Electrical Engineering and Music are two disciplines far apart from each other. People typically believe it is more likely for an electrical engineer to take on music than for musicians to take on engineering. Professor Yi-Wen Liu of Department of Electrical Engineering, National Tsing Hua University (NTHU) and Professor Yi-Hsuan Yang of AL Lab are two such examples of technical people taking on music. However, when I did my postdoc at Stanford in 1993, many of the postdocs and researchers had a prior degree in music. One example was Xavier Serra. Xavier is one of the founders of Vocaloid technology. He grew up playing cello and guitar, has a master’s degree in music, moved to the study of audio signal processing for music applications for his PHD, and now working on Computational models for the discovery of world's music.
The Fantasy of Music Signal Analysis
What does music have to do with AI? Think about the movie “2001: A Space Odyssey”. There was an AI computer for that sang a nursery rhyme along with music digitally composed by Max Matthew. For as long as the term AI existed, it had a close connection to music. AI made it possible to complete unfinished music creations by the likes of Bach, Mozart, Schubert, and Puccini.
What does music have to do with AI? Think about the movie “2001: A Space Odyssey”. There was an AI computer for that sang a nursery rhyme along with music digitally composed by Max Matthew. For as long as the term AI existed, it had a close connection to music. AI made it possible to complete unfinished music creations by the likes of Bach, Mozart, Schubert, and Puccini.
My SCREAM Lab at National Cheng Kung University (NCKU) made some small but meaningful progress on music score analysis in recent years. We explored the use of supervised learning to do music score segmentation and to determine performing style.
When we used Mozart and Haydn’s music as two different sets of training data, it was interesting to arrive at different results. The way they interpreted music differently reflected the differences in their music styles. We were fascinated by this finding. While there are much more and deeper studies to pursue, as of this year, this research finding has been incorporated into the techniques used by our virtual symphony orchestra.
Out of an obsession with sound and music
The initial idea of an augmented reality (AR) musical experience is represented by the virtual singer “Hatsune Miku”, but we wanted to take it a step further. Sometimes musicians need accompaniment for performances. Other times composers need an orchestra to help debut their work. Hiring an orchestra can be costly. A virtual orchestra fills that void.
When we used Mozart and Haydn’s music as two different sets of training data, it was interesting to arrive at different results. The way they interpreted music differently reflected the differences in their music styles. We were fascinated by this finding. While there are much more and deeper studies to pursue, as of this year, this research finding has been incorporated into the techniques used by our virtual symphony orchestra.
Out of an obsession with sound and music
The initial idea of an augmented reality (AR) musical experience is represented by the virtual singer “Hatsune Miku”, but we wanted to take it a step further. Sometimes musicians need accompaniment for performances. Other times composers need an orchestra to help debut their work. Hiring an orchestra can be costly. A virtual orchestra fills that void.
We started by having one cellist in our first performance. We are now able to present a string quartet. We expect to add woodwind and brass and have a full orchestra within a year. This is a multi-discipline collaboration of composers, instrumental performers, computer scientists, and animation artists. Such collaboration is common abroad; job and innovation opportunities are plentiful.
Programming: A modern bridge for multiple discipline studies
We often confine ourselves by our field of studies in Taiwan. That is not the case abroad. Anyone who analyzed Bach and Beethoven would tell you that these great musicians would have made good mathematicians.
Programing language and music are both languages. To me, the former is much easier.
A programing language has its grammar and logic. Compared to natural language and music, a programing language has fewer grammar rules, but the rules must be fully observed. Relatively speaking, a programing language also has simpler logic. One may combine multiple simple calculations to execute a complex calculation. Programming is the composition of grammar and logic to carry out a job. In programing there is no room for ambiguity. Well written and poorly written code yields the same output. For people who are well versed in music and natural languages, it is typically not difficult to pick up programing. I have students with a psychology background in my master program. They can pick up enough programing to accomplish a lot within a year.
Programming: A modern bridge for multiple discipline studies
We often confine ourselves by our field of studies in Taiwan. That is not the case abroad. Anyone who analyzed Bach and Beethoven would tell you that these great musicians would have made good mathematicians.
Programing language and music are both languages. To me, the former is much easier.
A programing language has its grammar and logic. Compared to natural language and music, a programing language has fewer grammar rules, but the rules must be fully observed. Relatively speaking, a programing language also has simpler logic. One may combine multiple simple calculations to execute a complex calculation. Programming is the composition of grammar and logic to carry out a job. In programing there is no room for ambiguity. Well written and poorly written code yields the same output. For people who are well versed in music and natural languages, it is typically not difficult to pick up programing. I have students with a psychology background in my master program. They can pick up enough programing to accomplish a lot within a year.
Since programing is a language, one would think it is better to learn a language sooner than later, but I think the difference is minor. For example, kids today start learning English at a younger age than I did and therefore speak better English than I do. However, although I have a heavy accent, my English is adequate to read technical papers, do some simple writing, and converse with foreigners; to me that is sufficient. If your interest is not in a technical field, then you don’t need to be an exquisite programmer. If you are a musician or artist, you only need to know enough coding to validate your creation.
Programming skill often empowers you to realize ideas. Picking up such skill before college is useful but it is not too late to wait till college. Do not stress over it. It is more important that you learn the skill for a right purpose. Likewise with learning English. You should learn it to use it, not to pass a test.
AI is a progressive field in universities. There are all kinds of classes on AI framework and algorithm, and interdisciplinary research topics are common. AI’s application in music is an example. The combination of technology, programing, and interdisciplinary knowledge is already powerful. With the addition of AI, one can do more and better.
Programming skill often empowers you to realize ideas. Picking up such skill before college is useful but it is not too late to wait till college. Do not stress over it. It is more important that you learn the skill for a right purpose. Likewise with learning English. You should learn it to use it, not to pass a test.
AI is a progressive field in universities. There are all kinds of classes on AI framework and algorithm, and interdisciplinary research topics are common. AI’s application in music is an example. The combination of technology, programing, and interdisciplinary knowledge is already powerful. With the addition of AI, one can do more and better.
“To be or not to be”
In Taiwan we tend to confine ourselves in our domain of studies and our comfort zones. It is a shame that students are good at taking exams but not actual practice, music majors must become musicians, hardware designers are inadequate at software, people who work with their hands feel inhibited to enrich themselves in theoretical subjects. Although we don’t have the resource at Junyi as they did when creating Hatsune Miku, we are proud to be able to demonstrate what technology can do for music in our virtual concert on March 12. Moving forward, with the help of technology, we hope to do more with music.
In this fast-paced world we should not confine ourselves to one career nor should we find our passion and curiosity only from school and work. One should constantly pursue new knowledge and learn new skills. I learned that from many of my peers and my students. Junyi students get more exposure to this notion than others. I trust you all can find your passion if you put your heart and mind to it.
Original article written in Chinese by Alvin Su, English translation by Angela Chin
Photo credit: Anita Hu, Junyi School of Innovation
In Taiwan we tend to confine ourselves in our domain of studies and our comfort zones. It is a shame that students are good at taking exams but not actual practice, music majors must become musicians, hardware designers are inadequate at software, people who work with their hands feel inhibited to enrich themselves in theoretical subjects. Although we don’t have the resource at Junyi as they did when creating Hatsune Miku, we are proud to be able to demonstrate what technology can do for music in our virtual concert on March 12. Moving forward, with the help of technology, we hope to do more with music.
In this fast-paced world we should not confine ourselves to one career nor should we find our passion and curiosity only from school and work. One should constantly pursue new knowledge and learn new skills. I learned that from many of my peers and my students. Junyi students get more exposure to this notion than others. I trust you all can find your passion if you put your heart and mind to it.
Original article written in Chinese by Alvin Su, English translation by Angela Chin
Photo credit: Anita Hu, Junyi School of Innovation
