Turning Data into Sound with Hannah Davis

I always describe data sonification as basically being just like data visualization, except instead of representing the data through visuals and bar charts, you're representing it through sound.

Hi, everyone. Welcome to a new episode. Welcome to another episode of data stories. My name is Enrico Bertini, and I am a professor at NYU in New York City, where I do research in data visualization.

That's right, and I'm Moritz Stefaner, and I'm an independent designer of data visualizations. And in fact, I work as a self employed truth and beauty operator out of my office here in the countryside in the north of Germany.

Yes, and on this podcast, we talk about data visualization, analysis, and generally the role data plays in our lives. And usually we do that together with a guest we invite on the show.

And that's the same today. But just before we start, very quick note, our podcast is listener supported. There are no ads. So if you do enjoy the show, please consider supporting us. You can do that with either recurring payments on patreon.com Datastories, or you can also send us one time donations on PayPal dot me Datastories.

Yes, and I also want to thank all those of you who are already subscribed on Patreon or sent us some donations via PayPal. This is so useful, and thanks so much. So, Moritz, I think you have a brief update from your side. You want to talk about one of your projects? You've been doing some climate visualization type of work.

Yes, and I also want to thank all those of you who are already subscribed on Patreon or sent us some donations via PayPal. This is so useful, and thanks so much. So, Moritz, I think you have a brief update from your side. You want to talk about one of your projects? You've been doing some climate visualization type of work.

Yeah, it's true. Not myself, but in fact I'm a freelancer. So usually I do projects, but I also teach occasionally workshops or in this case, a course in the digital media program at the Arts Academy in Bremen, where I live. And it's an interdisciplinary program between university and the art school, which is kind of nice. I always enjoy that. And in February, we looked at if we can find new ways to visualize climate change or anything around climate change. And the goal was really to go beyond heat maps alone, or make another super red map and think more about spatial or temporal ways, tangible ways to talk about climate change. And there were really, really cool results. Like the students had just a few weeks, but we had a few really cool projects. And I'll put the link in the show notes. We had data sculptures woven, interconnected with threads and meshes. Very nice. You could look at them from all sides, basically a big interconnected knotty mess. Or we had actual ice representing the change in extent of the arctic ice. So the students 3d printed molds and then made some real ice as an interface as well, and as a visualization. And there was also a sonification project. So Katja Striedelmeyer, one of my students, she was looking at the change when the apple trees start to bloom, and that's becoming earlier and earlier. And she made a little sonification of that data, which I really, really enjoyed because she also took an analog music box, like these little things with a, you know, that you can, how do you say, the handle you can spin and then it will pull through like sheets of paper. Yeah. And I can play. Yeah, I can play a little melody that represents the shifting apple blossom in Germany, if you want. Yeah. So the higher the pit shifts, the earlier in the year the apple blossom begins. And it's actually, it's a couple of days that it has changed over the last 30 years or so. So it's quite substantial, and then it begins again. So it's a loop. Anyways. Yeah. So I got thinking about sonification again, which is such a cool way to deal with data. And we did have an episode on sonification in the past.

You might remember one of my favorite. Yeah, yeah.

It was Scott Hughes, and he took the gravitational waves data.

Yes.

And directly turn it into sound. Super crazy.

Yeah. Yes. It was a funny one, and I think people really enjoyed it, and I think it's a perfect way to start the episode for today. Right. So we have on the show another person who is really an expert on sonification, and she's Anna Davis. Hey, Anna, welcome on the show.

Yeah. Yes. It was a funny one, and I think people really enjoyed it, and I think it's a perfect way to start the episode for today. Right. So we have on the show another person who is really an expert on sonification, and she's Anna Davis. Hey, Anna, welcome on the show.

Thanks so much for having me.

So can you briefly introduce yourself and tell our listeners what is your background and what kind of work you do?

Sure. So I'm Hannah Davis, and I'm a generative musician and researcher based in New York City. And I do many different types of projects in music and machine learning, subjective data and sonification. But basically, I think I can summarize my work by saying that I do artistic experiments with data.

That's cool. Yeah. And you have a lot of, it's not exclusively sonification work, but a big part of your work deals with turning data into sounds in some form, right?

Absolutely, yeah.

And I think you also have a quite unique approach there, which I found really interesting and which we'll hear more about in a minute. So just to lay out the field, some people might not even be familiar with the whole idea that you could turn data into sound and why that could be a good idea. So can you lay out the field a bit like, what is sonification? Are there different types? What is it particularly good for? Or how does it also relate to visuals? Maybe.

And I think you also have a quite unique approach there, which I found really interesting and which we'll hear more about in a minute. So just to lay out the field, some people might not even be familiar with the whole idea that you could turn data into sound and why that could be a good idea. So can you lay out the field a bit like, what is sonification? Are there different types? What is it particularly good for? Or how does it also relate to visuals? Maybe.

I always describe data sonification as basically being just like data visualization, except instead of representing the data through visuals and bar charts, you're representing it through sound. And to me, the way I generally think about it is that there's three main types of sonification. So that's oddification, which is directly converting data to sound, like, wait, in waveforms, just interpreting the data as amplitude over time.

So what Scott used it with the gravitational waves might be in that field where.

Exactly.

The data, the exact wave data, and just made it audible directly.

Exactly. And that can be really interesting. And I think that's where a lot of people get into sonification. The second one is kind of parameter mapping. So you do this in data visualization also, but it's mapping a value of the data to a component of the sound, where your focus is really on highlighting the data. And then the third one, which I'm mostly interested in and have been increasingly interested in, is music generation. So this is kind of the same as perimeter mapping, where you are still mapping values of the data to components of the sound, but the focus is not necessarily on showing the data clearly, but on instead creating interesting music. So it could also maybe have non data driven components. It could have more complex and hidden mappings. And, yeah, the point is really to create something that's interesting to listen to.

So if we think back to how it relates to visual visualization, maybe the parameter mapping is like if you do a really super clean Tufte esque chart.

Exactly.

Really think about, okay, what's the mapping between data and sound and no other decoration or no other texture?

Right, right.

And with the music generation, maybe you have a bit more liberties and you work a bit more on the emotional level.

Yes.

Maybe it's the sort of form of illustration almost, rather than just a pure.

Absolutely. Or art creation even.

Right, right.

But sound is really interesting because it has its own strengths compared to visualization, like the fact that it's temporal and that it is naturally multidimensional and you can carry a lot of data dimensions in it. It's great for pattern finding and cyclical data, and data that has these repetitive structures like you just played with the blossoms, because that creates really interesting, naturally repeating music. It's great for small units, for streams of data, and it's really unique. And then what's most important to me is that it's naturally emotional and it can move the listener.

Yeah.

It kind of, like, excites different parts of the brain, and it seems to be like, strongly connected to the emotional part.

Right, exactly. And very intuitively listen.

Right.

And it's like.

It's not the fact that it unfolds over time. It means that you. If you wanna experience it, you have to spend time.

Right, exactly. You kind of have to learn its own language, you know? Yeah, I like that.

Yeah. So, shall we listen to a few more examples?

Yeah.

What shall we play? I have a whole, like, browser here with lots of taps open. What should we go into?

Well, why don't we start with examples of what the three categories? Right. Well, maybe for modification, we already. I mean, our listeners can go back to our episode 75. Right. And here what we did with Scott Hughes.

I can briefly play one of these examples.

Okay, go ahead.

Yes. So it's two objects moving past each other in space. The gravitational waves, in this case, are loud when the small, but moves close to the large body, and they are quiet when the small body is far away. So you can learn about celestial bodies and how they relate to each other and what the effect is on the waves. I think we're close to a supernova type event.

Yes.

There we go.

Awesome.

And you really get a different sense of the forces. Right.

I think, yeah.

Yeah. So that would be. Yeah. Modification, directly taking wave data, turning it to something. Wavy sound. Wavy. Second, the parameter mapping. There's a great classic example from Amanda Cox and team at the New York Times where they looked at the results of different, like, race outcomes in sports. Right. At the. Must have been the Olympics or something. Do you remember, Hans, what it was?

Yeah. It's the speed at which Olympic athletes cross the finish line in relation to each other. And I love this piece because it comes with a visualization and you can see how close they are, like in little circles. But the sonification of it really does it justice. You really understand just how, you know, they're separated by milliseconds.

Right. Shall I play it?

Let's play.

Yeah, go ahead. So, first one is women's 1000 meters. Wow. Super close. Men's 1000, I guess. Yeah. Women's 500. Men's 500. Yeah.

Yeah, go ahead. So, first one is women's 1000 meters. Wow. Super close. Men's 1000, I guess. Yeah. Women's 500. Men's 500. Yeah.

It's amazing.

And what it does is also, in this case, it gives it the real scale. Like the chart could have any width, right?

Yes.

But the sound is like. How do you say that? It's like, on the actual scale, it's one to one.

Right. It's the actual temperature.

Right?

Yeah.

And that's super powerful. Yeah. And music generation. Yeah. I think. I guess then your work comes in. Right. Because that's what you do a lot.

That is mostly what I do. So this is my favorite music generation piece, which is by James Murphy from LCD Sound system. And he actually took tennis data and focused on creating interesting music from it. And so he's doing a multiple mappings. It's not necessarily a one to one mapping, like parameter mapping usually is, but he's using things like the court, the tennis score, the opponents, the temperature outside, to just create this kind of long piece. And it sounds really compelling.

So it's almost more like a portrait of the whole match, basically. Right.

Exactly, exactly.

Than this. Exactly. Data depiction. And he has cool sounds. He knows how to work his sounds.

Yeah, the samples are great.

I love it. Yeah, that's nice.

It's really beautiful.

It could totally be a fixed twin.

Yeah, yeah. That piece is really compelling. I think, you know, I always want to know what parts are influencing what, but I wasn't able to find anything about it. But I think that's okay. Just accept it for what it is.

But when we listen to your pieces, you can tell us what is what influences you. Right.

We will tell you tricks.

Exactly.

Yeah. So tell us a bit about your work. How did you get started and what are some of the key projects you have been working on?

Yeah. So tell us a bit about your work. How did you get started and what are some of the key projects you have been working on?

Sure. So I actually got interested in sonification through a data visualization class in grad school. And we did one week on data sonification, and it just stuck with me. And the first sonification I did was this sonification of different authors writing styles. If you want to play that, we can.

I could. Yeah.

So it was a couple different authors. And basically each note here is a syllable. The higher notes represent more descriptive words, like adverbs and adjectives. So this is Hemingway. I can already tell he has no descriptive words. I think this is David Foster Wallace, who just has these long run on, but really, really rhythmically beautiful sentences. And I actually loved that last one, which is an excerpt from Virginia Woolf's the Waves, which we'll come back to, because I recently did another piece based on that same text. So I basically worked with this project for maybe half a semester, and I found it just really interesting. But at the end of the day, I thought a lot of the pieces sounded the same. There are only so many variables that you can play with grammatically. And I started really thinking about why do people actually read books? And the obvious answer is that people read it for the emotions. And so I started getting really interested in seeing if, basically, I could translate emotions between mediums, if I could translate the emotions or the emotional data from the text into an emotional piece with the same kind of underlying feeling. And so that led to this project called trans prose, which I worked on for quite a while, years. And this was kind of the foundation of my sonification career. So I basically used what's called an emolex, an emotional lexicon, and which is a great name.

That alone is super cool.

I think its full name is actually word emotion association lexicon, but it's a little too long to say, but it's an amazing resource. It's basically 14,000 of the most common english words tagged with eight different emotions and then two emotional states, positive or negative.

That's great.

Tagged using Mechanical Turk. And so, basically, I would use this resource to go through and get these, you know, large, beautiful splines of emotions representing a piece of work, a novel. And then I would use that data and map it to different components of the music. And it took a lot of experimenting, and I tried a lot of things that didn't work, but most of the iterations ended up mapping the octave to, like, positive to negative ratios or joy to anger ratios. The pitches were basically based on an interesting map that I found worked but isn't particularly intuitive, which is that I mapped low emotional counts to more consonant notes and high emotional counts to more dissonant notes. And so what this actually ends up doing is that the plot of the novel is heard not in. In literal events, but in the emotional representation of events. And I really liked that mapping quite a bit and stuck with that for a while.

So it's almost like it could be a soundtrack while you're reading. It could be the fitting soundtrack to what's going on. Is that sort of the goal? More or less.

Not really, because I found that because you need kind of a long running average to have it be accurate. Basically, a novel could be reduced down to a piece that was, like, one to two minutes long.

Ah, true. Yeah. So the timescale is totally different.

Exactly. Exactly. I have done some more, like, real time pieces, but for this project, that's what worked.

Okay, cool. So shall we listen to a few examples?

Yeah.

Yeah. Shall we do Peter Pan first?

Yeah, that's one of my favorites. That's on the light end of the spectrum.

It's nice. It sounds very light footed.

It's cute. Yeah.

Yeah, exactly.

It's like a children's piece.

So remind me, the high pitches, what do they indicate?

High emotional count. So high emotional activity.

That was Peter Pan.

Hannah, can I ask you, I'm just curious, how do you decide which notes to play.

So that's the consonance dissonance mapping I made for myself. Basically, I took a whole scale and then rearranged it based on what I personally thought was most consonant to dissonant. And this is always a thing. I grappled with it a lot, especially in the beginning, because I wanted this objective mapping or representation of the feeling. But this just is what worked for me. I had tried so many other scales. I think the most naive approach that is kind of a common first approach is mapping low emotion counts to low notes on the scale and high emotion counts to high notes on the scale. But that doesn't actually sound interesting over time because then you just get a lot of low notes and occasional high notes, and they don't have any really musicality or music theory behind it, if that makes sense. It sounds a tiny bit random.

There's always the danger. Yes, it sounds a little bit random.

Exactly, exactly.

Because you still have to create something that makes sense.

Right, right, exactly. And I think I avoided randomness for a long time. None of the trans prose stuff has randomness in it at all. I've only just recently started to come around to it again. But it's very structured. Randomness.

Yeah, yeah. Let's hear a few more examples so we get a sense of the range, what's possible. Next one is the road. I haven't written that. I haven't read that book. What is it about?

Yeah, yeah. Let's hear a few more examples so we get a sense of the range, what's possible. Next one is the road. I haven't written that. I haven't read that book. What is it about?

It's a very dark post apocalyptic. Really, really dark. I think it's basically the story of a father guiding his son through the apocalypse.

Okay. Okay, I'll press play. We'll see what happens. Oh, that sounds dark.

This one is also not active at all, which you can also hear, I think.

Yeah, it's a tough read. This must be a tough read.

Right? Wow. Very dramatic end.

Right?

I think there is at some point. Yeah, yeah.

It sounds like at least in the middle. Yeah, something's happening. That's good.

There's some hope somewhere.

Yeah, I think so. Nice. Really makes me want to read the book, though.

That's not what most people say. And then this last one I really love because it's the story of a man doing really, really terrible things in a really happy tone. And I think the music captures that.

We can let our listeners guess. We play it first and guess what it is. We'll reveal it at the end.

Yep.

It's so fitting, actually.

Especially. Yeah.

Because Bach also plays a big role in the. Right. Yeah, yeah, yeah. And so it's totally. I don't know. It's a really good match. It's like March music as well, and.

Then it all comes together at the end. It really changes.

So that was a clockwork orange. Anthony Burgess.

Clockwork orange.

Pretty good match. Did you tweak that? So it fits a bit better?

So actually, the same algorithm made all three of those pieces.

Ah. So no interference from you.

No, there was interference at the beginning when I was just generally trying to find the boundaries for novels. But I've also done some other text projects with the same software, and I've basically had to change the mappings between mediums. So I did a piece based on news articles, and that was very different. So I changed it then I did. I sonified the 2016 presidential debates, and that was very different from novels, so I changed it then. But I was very curious in seeing if I could find this underlying true model, which I have since discarded.

Right. I mean, it's such a huge challenge to take something as fuzzy as a text and then. Yes, totally transplant it in this whole other domain and then using algorithms. I mean, it's an impossible task to start with. Right?

It's a lot.

Yeah. But I think it's amazing how far you got there and how interesting that also relates to the original work. I think that's. Thanks. Yeah, I love these types of reframings of existing stuff, if you think about it differently. It's so cool.

Totally.

Yeah. Yeah. So what? And so this. Yeah, you said it took you a while. It's all up on music from text. There's more, like different novels you have sonified. There's also videos that show you a bit how the exact mapping between the text and the sound is, which I found really helpful to understand what's going on. So you should definitely take a look there. So what are you up to now?

So right now I'm doing so many things. Well, I guess I'll talk about what I did recently, which was a really fun project for the synth beats laptop orchestra based in New York.

So right now I'm doing so many things. Well, I guess I'll talk about what I did recently, which was a really fun project for the synth beats laptop orchestra based in New York.

How many laptops are in this orchestra?

It was six for them.

That's nice.

Yeah, it was pretty amazing. It was my first time composing for laptop orchestra, and it was great.

Modern times.

Yeah, no, it was fascinating. I mean, I had known about there's plork, the Princeton Laptop Orchestra, and there's the Stanford laptop orchestra, but I hadn't realized there was one here too. But I actually, after a little while, actually, years of not working with grammar and text, I turned back to my first sonification, in a way, and created this piece called Percival. And this actually fleshes out that whole excerpt from Virginia Woolf's the Waves. And instead of, well, sorry. The computer basically creates an underlying track where each note is mapped to each word. So it's a very, very simple mapping. But then each laptop performer added one additional component. So one was the text conductor, just to move the text along and control the pacing and things like that. And then I had one person play a flute where there was dialogue. I had two people play elation and loss, basically being these real time sentiment taggers. So those were both mapped to corresponding notes to kind of create elation chords or lost chords. I had a one person play the triangle for percussion, basically on the syllables before interesting punctuation. And then I had just one person to be an emphasis on where I thought the text needed just a little boost. And so I think it worked. It could always do some workshopping, but it was really beautiful to see. And I liked seeing other people's interpretation of the same kind of components that the computer is usually interpreting. So it was a nice turn on its head in that regard.

Yeah. And what was the input for the performance? Was it the text or did they have a cue track of sorts?

No, it was the text.

The text itself. They would decide if something would fit into what. What they are supposed to do, basically.

Exactly. So, for example, the elation and lost people, I had them basically only press their, or only play their samples when they felt like the specific passage evoked elation or loss. And sometimes it would be at the same time, because it's an extremely beautiful, complex piece about this man who has just had a child and one of his very good friends has died. There's a lot of complicated and simultaneous emotions. And I've always found it really interesting.

Yeah. That makes so much sense because soundification is often just seen as this mechanical music production. Right. Or mechanical sound production, but music itself is performed. Right. So.

Absolutely.

Why not go all the way?

Exactly. And I agree. I'm trying to do much more. I think I end up talking about my work much more than I perform it or have it performed. And this year in particular, I'm trying to switch the stage pretty literally into doing more live stuff.

Little world tour.

I'm hoping to.

Let's listen to a bit of the Percival piece. Probably it's too long to play the whole one. I could imagine. That's great. It's really interesting to listen to.

Thank you.

And would the audience, would they also read the text or is it supposed to be like in parallel, like consume the text and the music.

Yes. I actually made a visualization where the text was coming up either one or two sentences at a time, and so the audience could read it along. Actually, when I hear it now, I was reading it much that I hear the word for each note.

Oh, yeah, that's nice. When it starts to tie together like that.

Yeah, I think it was important because that was the intention of the piece. And I think seeing how the text really does give it the structure, and especially with the pauses around punctuation, gives it this beautiful flow. I wanted the audience to be able to understand that also.

Yeah, that's great. Yeah. Maybe zooming out a bit. First of all, super fascinating work, and I really like the angle you take there with, like, okay, how can we work in the emotional expression of the whole thing and just take it just like making an audio chart, which is fine, too, but it's totally. I just love your approach there. So can you, like, you've been doing this for many years. Can you, like, let's say somebody gets started in the field, like, would be interested in experimenting a bit. Can you first maybe tell us a bit about, in your experience, the types of things that work versus that don't work so well or where you were challenged with and felt like, oh, this type of thing is hard to do in audio, and other things are nicer or easier.

Yeah, that's great. Yeah. Maybe zooming out a bit. First of all, super fascinating work, and I really like the angle you take there with, like, okay, how can we work in the emotional expression of the whole thing and just take it just like making an audio chart, which is fine, too, but it's totally. I just love your approach there. So can you, like, you've been doing this for many years. Can you, like, let's say somebody gets started in the field, like, would be interested in experimenting a bit. Can you first maybe tell us a bit about, in your experience, the types of things that work versus that don't work so well or where you were challenged with and felt like, oh, this type of thing is hard to do in audio, and other things are nicer or easier.

Yeah, of course.

Maybe how many things you can listen to in parallel or all these practical things.

Right, right. Those are all really good questions you can share.

We're happy.

The two things I always tell beginners to kind of avoid in the beginning is, are just really personal biases. There's no reason you actually have to avoid this. But I do say avoid odification because I think that it will limit you if that's where you start thinking about the mapping, because it's so literal. And good mappings, I think, really have to be thoughtful. You have to really think about what in your data maps to different emotional musical components, because every musical component conveys something different. You know, like low octaves definitely convey slowness or sometimes negativity. And you wouldn't want to map that to, like, I don't know, like, birth rates or something. You know what I mean? So really thinking carefully about your mappings is something I always say. And then similarly, I think it's good to start by mapping higher values in your data to higher notes on the scale and vice versa. But at some point, you generally want to move away from that unless you have a type of data that really works. So I think actually one area this really works is in climate change sonification. Wherever low notes do map pretty well to low temperatures and high notes map well to high temperatures, especially after they start getting to a certain level, you start hearing that shrillness and it kind of feels panicky in a sense, if that makes sense. I'm not sure.

Yeah.

But other things I would suggest, I always recommend getting your basic statistics of your data. I do a ton.

Start from the basics.

Right. I mean, some people don't do this, but I do a ton with mapping from the average. Maybe I'll only do something if it's one or two or three standard deviations away from the average. And that also creates this like, variability in your piece that makes it kind of not stagnant, I think. I didn't do that with a lot of the earlier pieces and they're less interesting for that reason.

So make sure you extract already a really strong signal before you even go to the sonification.

Yeah. Like have an understanding of the structure of your data. And that's also another thing I would say is usually data has so many structures that you can kind of pick and choose from. Like, you know, if you have, I mean, a lot of data has both an event and a time, and you can, you want to think about which one you want to use for the underlying structure. So, like, most of my work has this emotional underlying structure, but you could use time as an underlying structure. You could have. Sorry, when I say that, I mean that, you know, maybe 1 second will be one beat or one month will be one beat, depending on your data. But you could also have an event based underlying structure where each event is a beat and that changes the whole piece. And so thinking about what about your data is consistent enough to make it the structure of your musical piece is really important.

Right. So that's also the rhythm, basically, or the speed you set.

Yeah, the speed. Yeah.

Just have one x axis. That's. I just realized that.

Right, exactly.

Yeah.

So you really want to think about.

So you want to put it to good use.

Yeah, exactly. It's a good way to put it. I've never actually thought about it like that.

Yeah, I'm a visual guy.

Right. That makes sense.

These are great tips because I think, yeah, as you say, often people start with, oh, let's map all ip addresses to the sound spectrum. And then they are maybe a bit disappointed that it doesn't deliver like any patterns, but.

Exactly, exactly. You know, and so thinking about what.

You want to extract and how you represent. It is key.

Yeah, exactly. And all data does have a structure. That's the other thing I would say. Like the UFO pieces I sent in or I put in there were basically showing that point. Like you would kind of assume UFO sightings would be totally random data and not have any underlying context. But actually the underlying structure is that most UFO sightings are found at night. So you do get this kind of rise and fall in your musical pieces because there's nothing during the day, if that makes sense.

Yeah, yeah. Hannah, one thing I wanted to ask you is, so I think the same way in visualization, you can identify a number of visual channels that you can use to map properties of data to properties, visual properties. So say, I don't know, size, size, position, color and so on. So is there an equivalent when you are trying to build a music piece out of.

Yeah, yeah. Hannah, one thing I wanted to ask you is, so I think the same way in visualization, you can identify a number of visual channels that you can use to map properties of data to properties, visual properties. So say, I don't know, size, size, position, color and so on. So is there an equivalent when you are trying to build a music piece out of.

Absolutely, yes. I think the main ones are probably your key, particularly whether it's major or minor. Your instruments. I've worked a lot in piano because I want the melody itself to be emotional before I add instruments. But instrument choice in itself is a really good mapping. Volume is a really important one. Tempo and octave, are you vital? They changed the shape of the whole piece. The note length and all of the note lengths in relation to each other and the particular ones is a really interesting one.

So many options.

And of course they interact a lot, right?

Exactly, exactly. But you can get a ton out of just those on their own. You know, you can have really active low pieces or really slow low pieces, and those are totally different outcomes. And then also working with samples, I do a lot with midi and actually creating pieces that sound like actual instruments. But working with samples is really fun too, and can create their own kind of soundscapes.

Nice. So we should talk a little bit about how do you actually do that in practice? I'm sure some of our listeners would like to try it out, right?

Nice. So we should talk a little bit about how do you actually do that in practice? I'm sure some of our listeners would like to try it out, right?

Yeah.

In practice, when you actually want to realize it, what tools do you use and how does it look like? What are the steps there?

So for me, my favorite pipeline, I guess, is probably midiutil. I'm a python person, so that's kind of my go to in the past. For either Java or the older version of processing, I really recommend JFugue, which is a Java library that really has a ton of music theory and really solid documentation behind it. In JavaScript, I recommend Js Midgen to generate midi or if you want to work with samples, I basically learned programming on processing in P5, so I would recommend the P5 environment for working with samples, especially for real time sonification. A lot of what I do are full compositions where I generate the whole thing and wrap it in a sound font, and then it's a full piece. But I think a lot of people working with sonification do more real time stuff, and so I think P5 can be really exceptional for that. And then of course, you mentioned it's two tone. I haven't tried it out, but I've heard so many people say that it's a really exciting new direction.

Yeah, two tone you can also use if you don't know how to program, it's a browser based app, and you can take existing data and pick instruments and then say, how do you want to map the data to properties of that instrument, like the pitch or whatnot? And it's like in five minutes you can make a sonification. I think that's very cool. Yeah, it's a great start for sure. And then if you want to customize it more, you'll have to get probably into the programming side, but just play around and prototype. It's super cool.

It sounds good, at least to start thinking about it, because I think there's been definitely obstacles to sonification involving pipelines. So it's cool that that's out there now.

So I'm curious to hear, literally about the mechanics of how you do that. Right, so say, I guess you write some code, then at some point you'll have to listen to what you get out of it, right?

Yes.

And then I guess you need to go back and change what you don't like. Right. So can you briefly describe how this works?

So basically I have a python script that does all of the mappings and creates a midi with any of the libraries I just mentioned, usually midi Util. And then I have a little bash script that uses ffmpeg, a sound font, and just wraps it really quickly in a sound font and turns it into an mp3 or wav file. And then I can just double click that and listen to it. I see. It's pretty fast to generate any piece in python takes maybe half a second or less. So even though that sounds kind of like an annoying pipeline, it's not that horrible.

But it is hard to keep listening to sounds and variations of the same sound again. Right. It's difficult to at some point to judge anymore, like what even works.

Absolutely. Yeah. I did this one client project where, yeah, basically I think they gave me a day and I had been working on it, you know, the whole day, just nonstop on the same pieces. And I just, like, I sent it to them thinking it just was not good, and they loved it immediately. Oh, man. It's just so hard to tell.

Yeah. So this, I think now I'm curious to hear if you have one of those. If sometimes you have one of those creative missteps where you make a mistake, but the mistake is beautiful, so you decide to keep it there.

Yeah. So this, I think now I'm curious to hear if you have one of those. If sometimes you have one of those creative missteps where you make a mistake, but the mistake is beautiful, so you decide to keep it there.

Absolutely. Actually, I think in the last link I sent you, I'm actually recording this album called Siren Sailor sun, and I put a piece on there called Moby Dick, which was generated from Moby Dick, and there were actually two huge errors. The first was, like, I was off by half a measure in my calculation, but it sounds amazing. And then the second thing was that it was before I knew git, and so I actually lost the rest of the mapping that basically created my favorite piece from the. This time period. Oh, my God. Which is so sad. But at least I have the piece, so I would say it's even more.

Precious in a way, actually.

Totally. Yeah. It's, like, ethereal.

Yeah. Wow, this is fantastic. Yeah. I'm super excited now to dig through all this material. There's also a fun project, which is not like music generation, but more another creative way to working with sound that we could close with. Actually, it's the laughing room. Can you tell us a bit about that one? I think it's a good way to end the show.

So, this is a project from Johnny sun, who's a writer and comedian and a really funny Twitter person, and I. And basically, it's a room that was embedded with a neural network that was trained on the transcripts of stand up comedians, particularly women and people of color, to kind of, like, decrease the amount of sexism and racism that the algorithm would have. But basically, you would go into this room and you interact with your friends or anyone else who's in the room already. And the algorithm laughs when it kind of hears something worth laughing at. It was a very.

I can see all kinds of applications for that, obviously.

It was definitely the best project I did last year. Super fun.

Cool. I think we'll just play that on the way out. And before you hear it, we'll just say thank you so much to Hannah Davis.

Thank you so much for having me.

Thank you. I hope a few of our listeners are now really super psyched about sonification. Will send us all kinds of cool compositions in, like, three weeks, please. So thanks so much. And we'll. Yeah, you can find all of Hannah's work on her website. We'll put the link in the show note. It's www.hannahishere.com, if I'm correct.

Yep.

And we'll hear the laughing room to close. Thank you so much.

Bye. Thank you. Thanks so much.

Why does Darth Vader like his toast to be burnt on one side? Why? Because he prefers the dark side. APPlAUSE.

Why did Aquaman sleep in the nightlight? He was afraid of sea monsters under his bed.

Okay.

Is that funny?

That's pretty good, though the AI doesn't have a sense of humorous can and I have no humor. It was laughing at Helen's.

I don't know. And it was laughing at, like, socially.

Appropriate times, a little delayed. Maybe.

It understands this.

Hey, folks, thanks for listening to data stories again. Before you leave a few last notes, this show is now completely crowdfunded, so you can support us by going on Patreon. That's patreon.com Datastories. And if you can spend a couple of minutes rating us on iTunes, that would be extremely helpful for the show.

And here's also some information on the many ways you can get news directly from us. We are, of course, on twitter@twitter.com. Datastories. We have a Facebook page@Facebook.com. dot all in one word. And we also have a Slack channel where you can chat with us directly. And to sign up, you can go to our homepage datastory eas, and there is a button at the bottom of the page.

And we also have an email newsletter. So if you want to get news directly into your inbox and be notified whenever we publish an episode, you can go to our home page Datastories and look for the link you find at the bottom in the footer.

So one last thing we want to tell you is that we love to get in touch with our listeners, especially if you want to suggest a way to improve the show or amazing people you want us to invite or even projects you want us to talk about.

Yeah, absolutely. And don't hesitate to get in touch with us. It's always a great thing to hear from you. So see you next time, and thanks for listening to data stories.