Designing Exploratory Data Visualization Tools w/ Miriah Meyer

The intersection of interesting and computable.

Data stories is supported by Tableau software, helping people see and understand their data. Get answers from interactive dashboards. Wherever you go for your free trial, visit Tableau software@Tableau.com. Datastories. That's Tableau.com Datastories.

Hi, everyone. Data stories number 54. I'm Moritz. How's it going?

Hey, Enrico, how are you?

I'm good. Summer is here. Feels much better. And, yeah, teaching is over now. I can do some work.

Very nice.

Feels really good.

Yeah, yeah. I've seen. You've been a week to Israel, right?

Oh, yeah. Oh, my God. Israel was amazing. I've never been there before, and I was invited for a talk at this small conference called is Vis. Israel vis. And it's the first time, it's the first edition. Very interesting. A lot of different people with very different backgrounds. And, yeah, I loved it. And I loved, going around the country is amazing. Oh, my God. I've been to the Dead Sea, to Jerusalem, Tel Aviv. Beautiful.

Very good.

Beautiful. Yeah. I hope they do it again. I don't know if they can invite me again, but maybe they can invite you. Probably. Yes, as usual.

Very good.

And you?

Good. I had a little bee emergency today, so you know I'm keeping bees, right?

I know, I know.

Yeah. And so this is the time of the year when they, when they decide to swarm. Sometimes that means a big part of them flies off to a new place. And I was taking a little walk today, and I just.

So you have to catch them.

They gather already, like, outside one of the hives. I was like, oh, no need to get them. And so I put them into a bucket and gave them a new temporary home. And now when the call is finished, I need to go outside and check if they're still there, because they might be like, yeah, we don't like that we go somewhere else. Yeah. Let's see.

Nice.

Yeah, nice. That's my exciting news.

Okay, so we have another special guest today. We have Maria Meyer from University of Utah. Hi, Maria.

Hey, guys.

Hey, Miriah.

How are you? We are excited to have you here.

Well, I'm super excited to be on the show.

So we want to talk a little bit about all your work, but specifically about how to build visualization tools for scientists and researchers and what's the right way of doing that. I know you published quite a lot of work on the topic. So can you briefly introduce yourself so that our listeners know a little bit about yourself? What's your background, what you are doing in your research, etcetera.

Sure. So my background is very much rooted in a passion for science. I always love science and looking up at the stars with my dad. And I ended up being an astronomy major as an undergrad, in large part because it seemed like a really interesting way to study math and physics. But around the time that I was getting ready to graduate, I realized that I couldn't really imagine observing one wavelength of light for the rest of my life. But I really enjoyed building and making things. And so I was sort of exploring for a while and finally discovered computer science. And then I took a graphics and visualization course with Hanspeter Pfister a long time ago, way back before his Harvard days. And I really fell in love with the idea that in visualization you can be involved with science, but by building things instead. And so that's what prompted me to go to grad school for computer science. And so that's what I did. And then after I finished up, I did a postdoc, actually back with Hans Peter at his lab in Harvard. And I sort of used that as an opportunity to explore a bit and to try to really figure out what is it that I'm really, really passionate about. And that's when I started meeting a bunch of biologists in Boston and started seeing the kinds of tools they were using and realized that there was a huge opportunity to do something better. And so I started really exploring this idea of developing custom tools for small groups of people, and just sort of fell in love with that whole approach and the idea of getting to work with these amazing domain experts and learn about the cool stuff they were doing. And it was relatively successful because I ended up with a job after, which is what brought me out here to Salt Lake City.

Nice. So can you describe a little bit what kind of tools you built for scientists and how this works? Just a couple of examples so that people get an understanding of what it means to build visualization tools for scientists.

Nice. So can you describe a little bit what kind of tools you built for scientists and how this works? Just a couple of examples so that people get an understanding of what it means to build visualization tools for scientists.

Yeah.

So I think, sorry if I interrupt you, I think there are lots of people, especially those who are listening to this podcast, that see visualization mostly as a communication tool and as a way, the same way it's used in newspapers or, I don't know, by designers in general. There are lots of amazing examples, but I have a hunch that people are much less exposed to interactive visualization tools whose main purpose is to support exploratory data analysis, discovery, understanding and all the rest.

Yeah. So. Right. So the kinds of tools that I really focus on are the ones that are to help scientists answer their sort of basic questions and to really do this idea of a visual data analysis. And so most of these collaborations, they oftentimes start from the same place, which is some sort of, you know, scientists, let's take biologists who have just spent, you know, three or four years painstakingly in the lab collecting data, and now they're like, I have a whole bunch of data and I know there's something interesting in it. Can you visualize? It turns out that that's not a very useful place to start, but I guess we can get into that later. But so what I do is I spend a lot of time working with these scientists to really try to understand what are the kinds of questions that they have and how do we translate that into questions that we can, as computer scientists, that we can develop an interactive visual tool to help them understand. And I also, I think one of the things that makes these tools, well, there's many things that make these kinds of tools different from a communication tool, but I think one of them is that so much of the information that's required for a scientist to answer a question, it's very domain specific, and it's a lot of information that they just have in their head, things that they've spent many, many years studying and building up an intuition about. So I very much see these tools as sort of, as providing a blank canvas, this canvas that just contains some patterns that the scientists can then look at and bring to bear all that domain knowledge in order to really understand whatever it is that they're trying to study. So, yeah, these tools are supposed to be very open ended, very exploratory. They're not supposed to be taking any specific stance or perspective, but really trying to be this very open, blank page for scientists to bring their own knowledge to bear.

How specific are these tools? Is it, like, useful to ten people in the world or 100 or 1000? Like, what's the, how generic or how specific are they typically like the ones you've been working on?

Yeah, that's a good question. One, as a computer scientist, I kind of struggle with a little bit, but in large part, what I like to do is I like to develop tools for some. In the case of science would be like one lab. So, for example, one of the projects that I've worked on was working with a group that studied fruit flies. And in that lab there was about six people, and this lab was collecting data of a type that there was only one other lab in the world that was even collecting this data. So in that regard, it's a pretty, pretty small user group, and I think that this is really important for being able to ground our visualization, exploration and design in very specific tasks that we can then validate against, but also to know that we're designing things for real problems. And what I found over the years is that when you start to step back and reflect from these projects, that there's many things that do generalize that come out of these. Sometimes you'll develop a new visualization technique for a specific kind of data, but then later you're like, oh, there's this totally different kind of problem that has the same kind of structure and form. Oh, I could use this or some variation of that, or in the case of my sort of own things, what I'm really interested in is the idea of methodology and process. And so I oftentimes see these design studies as kind of a data driven approach to understanding, to better understanding. How do we do visualization? How do we do the visualization process itself? And so I'll reflect on many of these in order to see things that bubble up as salient and generalizable. But the short answer is oftentimes, most often it's ten or fewer.

That's great. I think it's so cool that these projects exist at all because I think quite often people assume it needs to be usable in such a wide range of settings, especially computer scientists, you know, usually strive for solutions that are so widely applicable that from my designer's perspective, a lot is lost. You know, all the benefits that come can come from a really, to the point bespoke design, right?

Yeah. And I have to say, I think, you know, I do have the luxury of being able to pursue this style of research, in part because of the kind of job I have. So, you know, I'm pretty fortunate that I, that I get to do that, but I think that these things do generalize and you can reach a wider audience over time.

So, Maria, in your experience, why so you've been working with quite a few different scientists, especially in biology. And in biology, I know, as a matter of fact, that there are lots of existing visualization tools that people use in the lab. And so why these tools are not. So why do they need specific tools if general purpose tools exist? Like, for instance, I know spot fire is very much used in biology, biochemistry, drug discovery, for instance. So in your experience, why they just cannot use them?

Yeah, I think that's a really interesting question. When I first started talking with biologists at the beginning of my postdoc, I mean, that was the thing. It was like, there's all these, like, great, you know, tools. There's many, many different software tools and visualization tools that have been designed for the bio community. But what these biologists, I kept hearing them tell me over and over again is that, sure, they could load their data into these general purpose tools, but that these tools were never answering their very specific questions. And so I sort of liken this to this notion of long tail science, where at the big part of the tail, you have 80% of the questions that 80% of people want to ask. And as you go out on the tail, it's those long tail specific questions that separate one biology lab from the other. And so if they don't have tools that help them answer their very specific questions, it's really hard for them to be able to do something different from the lab down the hall or the lab at the neighboring university. And so that's why I've been excited to focus more on that long tail side. And the biologists I've worked with, by and large, have been very excited and very receptive to this. And the tools that I've developed really have enabled my collaborators to answer questions that they really couldn't have done with the more general purpose tools. But with that said, I think part of, at least part of my own design process is the first step is oftentimes to throw their data into whatever we can, something like Tableau, something like Spotfire, just for, you know, for two reasons. One is for, well, largely for us, as the visualization designers, to start to understand a little bit more about their questions and about their data. But also, I think, to double check that one of these general purpose tools wouldn't just do the trick because you don't want to reinvent the wheel.

Yeah. That's so important. Yeah, it's exactly the same for me. And I know that Moritz, as well, uses Tableau all the time at the beginning of each project. Yeah, yeah. And of course, you don't want to reinvent the wheel. Right. We are much more interested in cases where we don't know that there's nothing available out there. But do you think that there is any long term solution to that? Or we will always be in a situation where very specific tools are needed and a visualization designer is needed to pair up with a scientist to solve this specific problem? Or is a solution, for instance, to make scientists more, I don't know, train these scientists in doing what we would do for them? What is the solution there, Enrico?

I imagine you have your own opinions on this, but I think what I imagine happening is, yes, I think that there will always be a need for very specific custom visualizations, but no, I don't. Well, and yes, I hope there's always a place in the world for people like us.

We would be needed for a while.

I feel like one of the big barriers is the fact that it's really hard to make a multiple view custom visualization tool that inherently requires programming and lots of tedious programming. If you look at some of the trends with things like D3 and processing, I think we're doing a really great job of creating these high level languages for creating a single view. In my own experience, where programming gets really complicated is when I have multiple views linked together in the interaction, your code becomes a nightmare. Right. And so I think there's some really interesting questions around how can we make that process easier so that someone like a biologist or a designer can create these sort of richer and more sophisticated tools, but without being a software engineer. However, I do think that there's this really specific view that we, as visualization practitioners and as computer scientists, take on the data that is very different than what most people who are entrenched in their own domain will think of. I think we're trained to think of things very abstractly and generally. And I find that in the collaborations I have, even just the conversations I have of me trying to understand their data and their mental models and putting it into my own framework, sometimes that can bring a lot to light for them because we're bringing this new perspective. And so I don't think that people like us are ever really going to go away. However, I think it could be more of this sort of emerging field of data science that once visualization and human computer interaction become more common within that community, I can see if we had easier to use tools that, that could really emerge as, you know, sort of this necessary collaboration with scientists.

Yeah, I think what is really challenging is to, is to acquire all these necessary skills because they come from very different areas. Right. So how do you, how do you get this knowledge? Because you need probably some computer science. You need to understand some statistics and a little bit of science, but also have interpersonal skills, be able to talk to people, which is pretty hard. Right. Especially when you're talking to a biologist. I had some interesting experiences in the past, and you have to have a little bit of a design touch because I've seen a lot of really ugly interfaces, and ugly interfaces don't work. Right. So how do you get to that. I'm sure that some of our listeners want to know, how do I actually learn? How do I become one of that? And it's hard because there's no single trace or path, right?

Correct. I totally agree. And to preference, I'm coming to this obviously from a very computer science perspective. So, like, more, I'm sure that your experience and many of the people like you, it would be very different. But I think within computer science, even within my department here, we're having a lot of these conversations around the fact that traditional computer science doesn't necessarily train people to do this kind of work. I had this very interesting experience this semester where I was teaching my first large undergrad class. It was our second semester intro to data structures and algorithms course. And as I was teaching it, I was thinking to myself, gosh, if I had to go through this program as an undergrad, I don't think I'd be a computer scientist. But, you know, and that really bothers me. Right. Because I, you know, department. And so I think that there's many people who don't have what we consider the traditional set of computer science or engineering skills that would do so well doing the kind of work we do. You know, I don't think you have to be an amazing programmer or, you know, know a lot about, you know, computing theory to do what we do really well. And in fact, you know, I'd argue that there's so many things you have to learn to do that, you know, you can only pick and choose. And so I'm really interested in an educational way and how we can open up computer science to maybe embrace a broader set of skills and of experiences. And how can we bring things like design thinking and design process into how we teach students and train them and also get them excited about what we do? How can we bring people that are really curious and engaged and do have these social skills and learn how to talk to each other? I think all these things are really important from an education perspective. And I'd like to think that computer science going forward is going to be broader than it currently is.

Yeah, I mean, a lot of it is really learning how to run a project, right? Like how to define a problem and how to approach potential solutions, how to try them out, how to evaluate them. And of course, you don't learn that at university necessarily.

Right.

But you did a lot of projects and you also documented a bit of the design approaches that worked and that didn't work work or that maybe were recurring patterns across all the projects you observed. So is there like a distilled version of an ideal project workflow you can describe, or is there something like a guideline you can give to people who don't know how to tackle a design solution?

I think for me, the short answer is to find people, to find a variety of people to work with on a team. So when I was first started doing this work in my postdoc, I had a very close collaborator who was a designer and medical illustrator by training and working with him alongside of some of the scientists who were very much engaged with what we were doing, was probably the most impactful thing that's happened to me in a very long time. I saw how he worked, and likewise, he also, I think, was very interested in the structure that I, as an engineer, would bring to bear on the problem. And so for me, the key is really to find a good group of varied people to work with. Barring that, read our design study methodology paper. I know it's a kind of cop out answer, but I think surrounding yourself with as many interesting and different people as you can to tackle a problem for me has worked really well, and it's not always easy. The way that I met this designer, his name is Bang Wong, and he works at the Broad Institute, was actually because at the time they had the Broad Institute, which is a large biology center in Cambridge, Massachusetts. At the time, they had this artist in residence named Danyel Cohn. And Danyel was really interested in visualization, as this word is very broadly construed. And so Danyel was just a connector. And so he just brought together people from Mitzvah and from Harvard and from different places around Boston, people with incredibly varied backgrounds. I mean, there was like me, I was like the most visualization kind of visual, visual person there. But there was people from the media lab, there was some biologists who were interested. There was artists like Danyel. And he just brought us together to, like, think about crazy ways to imagine that we could turn genomics data into visual things. He talked about 3d windows and all kinds of stuff that at the time, I was just like, oh, my gosh, this is not going anywhere. What is he talking about? But I think that experience of being confronted with this very different view of the world, this very different perspective, and trying very hard to put myself in his shoes and understand where he was coming from, I think I grew a lot in that and through those experiences.

I think for me, the short answer is to find people, to find a variety of people to work with on a team. So when I was first started doing this work in my postdoc, I had a very close collaborator who was a designer and medical illustrator by training and working with him alongside of some of the scientists who were very much engaged with what we were doing, was probably the most impactful thing that's happened to me in a very long time. I saw how he worked, and likewise, he also, I think, was very interested in the structure that I, as an engineer, would bring to bear on the problem. And so for me, the key is really to find a good group of varied people to work with. Barring that, read our design study methodology paper. I know it's a kind of cop out answer, but I think surrounding yourself with as many interesting and different people as you can to tackle a problem for me has worked really well, and it's not always easy. The way that I met this designer, his name is Bang Wong, and he works at the Broad Institute, was actually because at the time they had the Broad Institute, which is a large biology center in Cambridge, Massachusetts. At the time, they had this artist in residence named Danyel Cohn. And Danyel was really interested in visualization, as this word is very broadly construed. And so Danyel was just a connector. And so he just brought together people from Mitzvah and from Harvard and from different places around Boston, people with incredibly varied backgrounds. I mean, there was like me, I was like the most visualization kind of visual, visual person there. But there was people from the media lab, there was some biologists who were interested. There was artists like Danyel. And he just brought us together to, like, think about crazy ways to imagine that we could turn genomics data into visual things. He talked about 3d windows and all kinds of stuff that at the time, I was just like, oh, my gosh, this is not going anywhere. What is he talking about? But I think that experience of being confronted with this very different view of the world, this very different perspective, and trying very hard to put myself in his shoes and understand where he was coming from, I think I grew a lot in that and through those experiences.

So can you tell us, so you mentioned a little bit of your design study methodology. Can you describe more in details what it's about?

Sure. So this was a project with Tamara Munzner and Michael Sedelmayr. And we just got to the point where as a group, we had done, I don't know, 20 some design studies, and we felt like there was probably something that we could reflect on and put into words to help provide some guidance to other people who want to do this kind of work. And so the result of it was, was a framework that broke up the process into multiple steps. I think for me, the most important parts of that framework are the fact that there's these three stages. The first stage is all the stuff you have to do before you start a project. And here it's recognizing that you need to learn about the space of visualization techniques and methods, because if you only know about a node link diagram, you're going to be pretty limited in how you can, say, visualize a graph, for example. And so it's really important to get that breadth of knowledge. And also how do you find good collaborators and make sure you're talking to the right people. And then the sort of middle, the middle part of the framework is this very iterative design process where you go and you talk to people, you try to understand what they're doing, you do some prototyping, you get feedback, and you sort of work your way towards a final tool. And then the end of it is this reflection stage where I think this is what's really important for giving back to the visualization community, is reflecting and saying, well, what's generalizable about what I've just done? And so that at a high level is what the methodology is about and.

The design study, is this the actual project or is the design study the documentation of the actual project? I'm just not clear.

Like a case study. Yeah. So the way that we define design study in our paper on this is that it's a project that tackles a real world problem. So it requires talking to real people and having real data where you design a solution. So you try a lot of different ideas and you winner those down to a final tool that you deploy out into the wild and then go and get feedback and understand how effective has your solution been? Are there things I can improve on and things like that? So it's this very problem driven approach to a specific project.

So at the beginning you said that you need to understand what is a good collaborator, what is a bad collaborator. I'm sure you had some.

I had quite a few, yeah. Yeah. You know, this is where I feel really fortunate that there's way more people that want to collaborate with people like us than there are people like us. So we can really pick and choose. Yeah, I think one of the first classes of collaborators that probably won't work out are the people who just view us as a software developer, you know, who are like, well, I have this really, really awesome, interesting data, and I just need you to implement an interactive heat map for me that does blah, blah, blah, blah, blah.

Everyone has this problem.

Yeah, it's like finding, you know, it's for people who can't, who aren't willing to get to the point where they understand what visualization research is and even just the visualization design process, and that even though they're the domain expert, that we are experts in something different. So for me, that's always like the first red flag and there's lots of other red flags. So one of the absolute requirements for a design study for me is that you have to have real data. It can't be sort of the promise of real data. With that said, I've had some recent projects and one project recently with a student working with army research lab where the data was confidential and are classified. And so we had to think a lot about, well, what does that mean for, does that mean we just can't do a design study? And that wasn't really, you know, we weren't going to accept that as an option. And so we, you know, we sort of developed some strategies for how we could work with the, the sort of toy test data that they gave us and questions that we needed to make sure we could answer in order to understand the scalability of the data, where the problems would be. And so, anyway, so, yeah, so we developed some strategies for how to deal with that. So I wouldn't say you absolutely can't have real data, but if you don't, you need to really understand what's going to be different about the real data. So those are some of the things that, you know, we always look out for. And then, honestly, design studies, I think, take so much time in interacting with those collaborators is that you have to, have to like them. Yeah, you have to be really cool.

That's a good thing. You get along on a personal level before you start anything substantial together.

I mean, honestly, like, for me, more oftentimes more important than the actual space of the problem itself is, do I want to work with this person? And if so, then we'll find something.

Yeah, yeah, yeah, that's a good point. Data stories is supported by Tableau software, helping people see and understand their data. Tableau lets people connect to any kind of data and visualize it on the fly. Databases, spreadsheets and even big data sources are easily combined into interactive visualizations, reports and dashboards.

Yeah, yeah, yeah, that's a good point. Data stories is supported by Tableau software, helping people see and understand their data. Tableau lets people connect to any kind of data and visualize it on the fly. Databases, spreadsheets and even big data sources are easily combined into interactive visualizations, reports and dashboards.

And by now there's a new version out. So the latest version is Tableau nine. And in Tableau nine you'll find features that makes the product smarter about what you're doing from a new start experience with data prep tools to more analytics features and smart maps, for instance with geographic search. So you can just type in the name of a city directly, go there. Really nice. Across the entire analytical flow, they have invested heavily in performance and so everything's much faster now. And there's new features to help you share your findings and also collaborate with data. The thing I really like the most is the new data import tool, because you can finally split individual columns by delimiters and also pivot directly a data table. So what you know from Excel, the pivot function is now directly into data import, and that saves so much time. So I'm a big fan of that.

Great. So if you want a free trial, visit Tableau software, Tableau.com Datastories. This is Tableau.com Datastories.

Great. So if you want a free trial, visit Tableau software, Tableau.com Datastories. This is Tableau.com Datastories.

That's right. Now back to the interview. Here's another thing. So I work a bit with scientists too, and there's a couple of problems I keep running into, probably from another angle, because I'm more perceived as the designer, of course. But one recurring problem is this thing that there's a lot of trust in default visualizations or in learned idioms, and there's a big fear of receiving just bad reviews for non standard graphics. So each scientific field I have encountered has a standard way of plotting a certain data set. Usually it's bad, it's like jet color palette from MatLab and like 3d manifolds.

I totally get it.

Do you have a technique for, I don't know, for getting people to a point where they accept that innovation is, well, can be helpful in that area? Or how do you deal with that problem? Have you encountered it? Or how do you deal with it?

Yeah, I had one project. Again, this is going back to this fruit fly project where the lab I was working with, we developed a tool, published a paper, and then they went off and did their analysis. And so then when they were publishing their, their scientific paper, one of the co authors was this guy named Michael Eisen, who is sort of known as like the father of the heat map within the biology community. And in this tool, what we had eventually done is gone away from a heat map for looking at this gene expression data that's almost always shown in the heat map to doing these little curves and stuff. And so I remember that the PI I was working with, when she was creating the figures, she wanted to use these curves instead of. And she told me later that she was really nervous when she showed the diagrams to Michael Eisen. And he was just like, oh, yeah, this is clearly the way to show it. So for me, this was like this huge, like, this was just such a huge success. I was so excited. But with that project, it was really interesting because the first prototype that I created for them, for looking at this data was to use heat maps, just like they were doing using the rainbow color map. You know, all these things that we as visualization designers know are not ideal. And so what I found, and not just in that project, but many projects, is that I don't just dump on them these brand new, totally out of the blue visualizations, but it's like this slow transitional process where they slowly build up trust. You change little things at a time. So, like with these heat maps, for example, the first thing I did was I added these little overlay curves on the side where she was like, oh, yeah, that's really great, because you can see trends. I was like, yeah, but now we're doing color encoding for one and positional encoding for another. What if we just change this color encoding to these curves? She was like, oh, yeah, but maybe you could give me a button so I can go back and forth. And after a time, she stopped going back and forth. Yeah. And so for me, that was a really good learning experience in this process of slowly gaining trust. And I think it's also an opportunity for us to educate the people we work with about, like, well, I was thinking of making this change because of these studies that have shown whatever. And in general, I think scientists, again, if you have good collaborators, they're going to be open to those kinds of things.

Yeah, I think you're raising a really good point. And my experience is very similar, and I think it just doesn't work. If you go there and say whatever you've done so far, it's just plain wrong. You come with this huge new block of knowledge. And I think what I've learned throughout the years is also to respect more some practices that have been established over many, many years. Because, I mean, my feeling is that most of probably these scientists are really clever people. If they are using something for many years, there must be something good going on there as well. It depends. It depends there's really both. It depends. Of course, this is not always true, but I think I learned to be a little bit more careful in criticizing everything. And I think in visualization, as well as in many other fields, we have our own dogma. Right. And we believe a lot of things and they are not always questioned. So I think it's important to remain open all the time.

That's true, absolutely.

And I think this gets back even to your comment about the necessary sort of interpersonal skills for doing this kind of work, this recognition that you're a team, you know? And sure, we each have our own expertise, but we can't completely ignore what the other people on the team are, you know, what they feel strongly about.

Yeah.

And that they also operate in a certain context. We can say, like, yeah, we should use, I don't know, circles, but they operate different context. That's super important. That's absolutely true. Yeah. But Enrico, I think there's also a few scientific cliches in visualization, like in specific sciences that are just plain wrong. They're just inefficient. They could be much better. And just people don't realize it because they never made that switch to a new form of visualization.

Well, I also think that this could be my just limited foresight because I've only been doing this for x number of years. The data is just getting more and more complex. And so there could have been these visual conventions that worked back when it took ten years to sync with a single genome. But now that you can do it in, whatever, less than a day, suddenly you're confronted with a scale and complexity of data that they didn't have before. And so it could be that people are starting to come up against this and there's just a need to revisit those conventions and to think more broadly about how to change them.

Yeah. And I have to say another thing in my experience is that people, if, if the people you are collaborating with never get excited about what you are doing. It's a big alarm because visualization has this big power of this huge wow effect. Right. And if you don't get to that, at least at some point, I think that's, that's a sign that something is going wrong. And I had this thing myself a few years back and I, and I realized that something was going wrong and it was going wrong. So, I mean, I think this is pretty, pretty, a pretty unique feature of visualization, that when it works, people are really like, they love it, right? They want to hug you. So if this doesn't necessarily even do.

Yeah, it's great to be the hero in every project, right?

But you're right. If, if you're working together for half a year and it never comes up, somebody learns something new out of your visualization. You're absolutely right, Enrico. Then something's wrong. And then we need to be honest as well and say, like, oh, probably our approach didn't work here, or at least ask, like, what's the problem?

Yeah, exactly. Exactly. So, and Maria, regarding the iterative process that you have in the middle, I'm curious to hear from you, what's your take on prototyping? How does prototyping work there, whether we have good methods and tools for prototyping? Because I'll tell you how it happens in my lab. My students are just amazing programmers and they are able to come up with amazing stuff in 24 hours. But we cannot pretend everyone to be like that. I am not like that. Right. So what do you think about that?

Yeah, exactly. Exactly. So, and Maria, regarding the iterative process that you have in the middle, I'm curious to hear from you, what's your take on prototyping? How does prototyping work there, whether we have good methods and tools for prototyping? Because I'll tell you how it happens in my lab. My students are just amazing programmers and they are able to come up with amazing stuff in 24 hours. But we cannot pretend everyone to be like that. I am not like that. Right. So what do you think about that?

Gosh, it's almost like I would have given you these questions to ask.

I do a lot of paper sketching, for instance.

Yeah. So, yeah, in my group we talk a lot about this idea of rapid, iterative prototyping. This is one of those things that we believe very deeply is incredibly important. And yet I think that in general, things go great until all of a sudden you have data and then it's just like, well, great. You know, I can sketch on paper, I can do all kinds of stuff, but now I have this data and, you know, I think we also, there's also, I think, an understanding that when you prototype, you need to prototype with the real data, because oftentimes if you don't, it breaks your.

Yeah, absolutely.

Whatever it is you've created the first time you look at it. So this is one of the things that I actually have a project that's in collaboration with some folks at Microsoft Research. We've been very curious about this. And so we actually did some work where we were trying to understand how designers worked with data. Here's a group of people that basically their bread and butter is prototyping. And so my colleague Danielle Fisher and I started having these conversations with each other about, wow, how did designers do this? And we had this hunch that they weren't using a lot of the tools that we as the visualization community were building. Like, why is that? And then we sort of started to learn that really what they were largely doing is manually encoding data in illustrator, like where they draw axes and tick marks and then draw bars of like an appropriate size. And then we were just sort of floored, like, what would make people do this like an animal? It was just unfathomable to us anyway. So we tried to recognize that, you know, there may be something here we're not quite understanding. And so we did a series of interviews with some design professionals that we knew. We also did some, well, we did some interviews to try to find out what was their process, how did they work with data? How did data change things or not? We also conducted some controlled studies where we brought design students into the lab. We gave them a data set that we had. They spiked with some outliers and like, you know, some task about creating an infographic. And then we just observed how they, what they did. And then we also did some observation of a hackathon that involved designers. And from all this, we, you know, we were able to come up with, you know, a series of patterns that we saw things of how these designers were working and some of the things that were really interesting to us. One was that, yeah, this manual encoding is super common, but it's not necessarily always a bad thing. So the manual encoding was often the point where the designers would get into their data. That was sort of their data exploration phase. And we also saw a lot of times that people did, in fact, use tools like Excel or Tableau or even writing simple processing scripts, but every single time they used an external tool, they always brought the final design, they always brought their visualization into illustrator to do their final design refinements. And the thing that was most poignant to us from all of this was the fact that once they brought things into illustrator or even manually encoded data, there was no going back. They were not able to accommodate data changes or even being able to say, oh, hey, I'm looking at this as a node link diagram. Why don't I try an adjacency matrix? These kinds of changes were incredibly hard. And so this got us really interested in this problem of how do we create a more flexible ecosystem of tools? That's currently one of my students dissertation projects.

That's a really hot topic, like, how do you have tools that allow this fluid, parallel exploration of data and concept spaces and shapes and, you know, encodings and layouts and so on?

Right.

What do you think of Lyra? Have you played with Lyra?

A little bit. I've played with it a little bit, and I think it does some things really, really well. You know, it tries to make this process of building up data or building up a visualization based upon data, very efficient, and there's a lot of flexibility there. But we saw that there's still a lot of the designers we talked to and a lot of the infographics we looked at. People are wanting to do some pretty unique stuff. And one of the things we heard from our designers, reason that they didn't like the sorts of tools we've created for them, sort of engineering community, is that they didn't want their stuff to look like the next person's, you know, they want their. Their visualizations to be unique. And that's why they really like illustrator for its richness there. So the philosophy that we've basically been working under and are exploring right now is this idea that, you know, there's been a lot of work and effort and deep thought that's gone into a tool like illustrator and a tool like Tableau and a tool like Lyrae. Building a monolithic tool that's going to replace everything seems kind of silly, and certainly my student will not finish it in his dissertation. So instead, we've been thinking about this idea of how do we bridge between all these different tools that exist? What are the things that you need to do and keep track of and take care of in order to support iteration between the tools that exist out there? So that's what we're working on right now.

Oh, yeah, absolutely. I think there is an interesting ecology of tools out there, but still it's not clear how to move from one to another.

Yes, I agree.

I'm really curious to see what is going to happen in the next few years in this space, because there is still a lot to do. There are some amazing tools out there, but I also believe there is space for some other tools.

Yeah, the whole question, how we use the tools is actually much more interesting. This is exactly the way, of course, you know, your work fits in. Like, how are the workflows? Like, what should we start with? Like, which questions do you want to get answered when and which can be postponed to a later stage? And how do you resolve blocks in a project workflow and these types of things? I think that's super fascinating and it's not investigated much. And the paper, I found it really the reflections on how designers design with data. I found it really nice to read and also very entertaining because it's a bit like somebody observes this alien race, what they do, they encountered some data. How did they react to it? That was very fun to read, actually.

Yeah, the whole question, how we use the tools is actually much more interesting. This is exactly the way, of course, you know, your work fits in. Like, how are the workflows? Like, what should we start with? Like, which questions do you want to get answered when and which can be postponed to a later stage? And how do you resolve blocks in a project workflow and these types of things? I think that's super fascinating and it's not investigated much. And the paper, I found it really the reflections on how designers design with data. I found it really nice to read and also very entertaining because it's a bit like somebody observes this alien race, what they do, they encountered some data. How did they react to it? That was very fun to read, actually.

So is there anything in there that you felt we didn't quite get right?

No, I share the observations, and I think you described the gap that needs to be bridged quite well, that there is some really nice, very manual procedural practice in design, but how to connect that to active data exploration is the big problem at the moment. Some people are just personally better at that than others, but there is still a big gap in general from both ends.

I think this gets back to what we were talking about earlier, too, Enrico, when you brought up this, you know, are we going to put ourselves out of a job one day? But I think it's, you know, it's this, even this middle step, you know, the prototyping that we all struggle with, you know, sketching with data. But I think, you know, once we do explore the space and get better tools in place, it's going to open this up to a lot more people. I don't think this necessarily means that every biologist and every physicist and every geographer is going to have the knowledge or the desire or the time to learn what they need to learn about how to visually represent things, but I think it will. There will be a broader class of people, designers, journalists, you know, data scientists, who will have access to creating these kinds of things without having to learn how to program D3. Yeah.

Oh, yeah, absolutely. I think that's very much needed. And, yeah, I was briefly. We. Me and Moritz were briefly commenting that before the interview, that it's surprising nobody did that before. Some. Some study like yours, on understanding how designers do what they do. And it's surprising when you look around to see how many things we don't understand very well yet. I mean, even. Yeah. Last week or two weeks ago, when I was in Israel, I was, in my talk, I introduced. I had one slide saying, we don't really know exactly how people read this stuff once it's published on the web. Right. We don't even know whether people do read them or not. Right. And that's pretty crazy.

Yeah. I mean, too often. And that was feedback. I always got it from these. These scientific tools I created, too, where people are always, like, surprised, like, oh, your collaborators are using this in their lab? And I'm like, but of course, like, what. What other measure of success would you have? But, you know, unfortunately, too, I think, at least within the academic world, the way it's set up doesn't necessarily encourage us to do this longitudinal type of follow up work, you know?

Like, yeah, absolutely. Yeah, that's a huge problem for me. Yeah. Yeah. Because we don't have the right incentives to do that.

Right, right.

So I love to do this kind of work, but after some, some time, it's just no longer worth it, at least for my career. So I have to sacrifice something else in order to go deeper into this direction. And I think it's not, it's not easy. And I, I think Tamara did something recently. I don't know if you were involved in that, on a very long term kind of adoption study, is that correct? Yeah, I'm talking about Tamara Munzner.

Right.

But it's very hard. So I'm wondering if you have any experience with observing, let's say, adoption and the use of any of your tools or other tools on an extended, that time period, because that's very hard.

Yeah. So. Right. Like you, I wish this was something I could say. Oh, yeah. I have lots of experience doing this. I do have one project right now where we are planning a series of more longitudinal types of things to do. And, you know, it's been a bit interesting for me in figuring out, like, well, where would we say, publish that? Or what's the community we would target? And in some ways, we're targeting the domain a little bit more. But it's just such an interesting project that we're all willing to do this and keep going. And you know what? Like, if it doesn't count as a, quote unquote, computer science publication, whatever.

Yeah, yeah, yeah. Of course. Yeah.

But, yeah, no, I think I don't have a lot of experience, and it's something that I do regret and I wish I did.

Yeah. But it's hard.

It is hard.

There's one project I want to learn more about. You briefly told us about that you're working with poets in a new project. Can you tell us a bit? I know it's a preliminary sound.

Yeah, that's actually, that's the project that we're going to do some longitudinal work on. Yeah. So this is one of my most interesting projects right now. And so this is working with some local poets who got very interested a couple of years ago in the idea of how technology can influence their experience as poetry scholars and poets. And it was actually they first worked with Min Chen, who's now at Oxford. He was the one that did the hard, he was the one that did the hard work of convincing them that they should care about technology, because they're like, I don't want anything to get between me and my poem on a piece of paper. In fact, this is one of the things that's so interesting about this project is the fact that these poets and their larger community are openly resistant to technology. And so one of the things we've had to figure out, they say over and over, oh, we don't want a computer to solve the poem. That's what we do. That's the fun part. That's not what we want to do. So we've had to really grapple with this idea of, well, what is it that a computer then does? What does a visualization do? And we've. I'm becoming more and more aware that it's about spurring ideas and this notion of creativity, that it's there to augment what they do but not replace what they do. And so this has been, gosh, I guess we're over two years now in this collaboration. We probably spent the first year and a half figuring out, well, what is it? We're even going to datafy in a poem, right? It was like this. We had to find the intersection of interesting and computable. Used to always say things like, oh, we want you to detect metaphor and show it to us. And I was like, wow, if you could detect metaphor, I think you'd win a Turing prize. So I think it's a really hard problem and we're not going to tackle that. But we finally figured out that sound was very, very interesting to these poets. And sound, with some caveats, is very computable. So I have a PhD student, Nina McCurdy, working on this, and she's just done an amazing job of collaborating with these poets and really understanding their sensibility and trying to really figure out, like, what about sound is interesting? How do we give that to them in a way that's not overwhelming? And so we've had a couple of papers now, one coming out and one under review on this. And it's just so interesting because, you know, we struggle with a lot of things that I wasn't prepared to struggle with, things that, that are kind of opposite from working with scientists. So, for example, anytime we say the word uncertainty or ambiguity to these poets, they get so excited. Whereas I feel like with scientists, uncertainty basically is just something that gets in the way of them finding the answers to their questions. It makes it more confusing. But for these poets, they're just like ambiguity. Oh, that's the best we've had to learn to embrace some of these things that I feel like I've been trained not to embrace. But then also evaluation has been really, really challenging because these poets, anything we give them, they're just, a, they're so excited about, and b, it just leads to insights and so insights are abundant, and so that's not really useful either. And so we're really trying to figure out how do we, how do we measure our success with this project? How can we figure out a, which direction to go next?

How do you end up today if they write better poems? That's the metric. Right, but how do you measure, how.

Do you measure that?

Yeah, well, exactly. And it turns out that they are actually using our tool to either refine or to write new poems, which was not something we actually expected them to do. We were thinking, oh, we're going to show them sound and they're going to analyze how sound evolves throughout the poem, but no, some of them are using it to create new poems. And so right now, what we're thinking about are a series of experiments that we can do to try to get at this notion of creativity and how these poets are using the tool and what it means for the potential role of technology within poetry.

So just to be clear, do you sonify the things they write or is it, what's the data? What's the data?

Basically, you can load in. Is it text file?

Yeah. And then you make a sonification of the text file. And that helps them spot patterns in the text they hadn't seen before.

Exactly, yeah. Specifically, we detect rhyme. It turns out rhyme is incredibly broad and fuzzy. Like, there is no definition of rhyme. And so one of the things we did is we developed a formal language to describe rhyme in many, many forms, so that once you describe a rhyming pattern, you can then find it and then let them and explore intersections of patterns and things. It's pretty cool. We'll see if we have a page.

That's good. You should put that online somewhere so you can experiment yourself and see how you tweak it.

As soon as it's no longer under review, we will release it.

Sounds really good. Do you also play with the rhythm of language?

We don't right now, but that's something that my student who's working on this, she's very interested in hip hop, and so she's been really interested in that as well. But we haven't yet explored how to integrate that yet. But I think it would be very interesting.

We need to do a lot, clearly, in a year when the hip hop I would love.

That's just perfect. You know, we always get a lot of criticism because in a podcast, you cannot, you cannot really do a podcast about visualization. Right.

But sonification, that might be. Sounds doable.

So what is the visualization part in this project? What do you visualize exactly?

Yeah, I have to say the actual visualization itself is the least interesting part of this project. But I will say I feel like this process that we as visualization designers go through is what we've been doing. This idea of a design study of spending a lot of time talking to our collaborators, trying to figure out how they view the world and what's interesting to them. And then in this case, once we could figure out what the data meant, it took us a long time to develop a system to pull out the data. But the visualization problem is that I think it's kind of interesting. They were really adamant about anything that we show them being in the context of the poem, just because, again, you know, we're not solving the poem, we're giving them. We're giving. We're showing them things to help them better explore, understand, or come up with new ideas. And so there's so much going on in a poem that we're not datafying, that we had to show the data in that context, whatever we show. And so we have this notion of poem space. You know, you can think of it as a 2d space, but I think with some interesting things that you always read, well, at least in English, you always read left to right, top to bottom, and that constrains the visualization. We came up with somewhat. And so we have a multiple link view system to help them get in and see different things. But, yeah, so it was really about how do we show these different sets of words, and particularly how some words sort of act as hubs, where a lot of different writing sets are coming together and then things diverge. Sometimes they overlap. Those are the kinds of things that they were really interested in. So it's kind of a set visualization problem, a little bit of a graph visualization problem, a little bit of a 2d spatial problem all rolled out.

A little bit of everything.

Yeah.

Nice.

There's some ambiguity thrown in there too, of course, just for fun.

It's a rhymes.

So I want to ask you one last thing. You've been mentioning a few times, data science. And I'm curious to hear from you because there are quite a few data scientists listening to the show. What's your take on what's the role of visualization in data science?

Super important.

Yeah, yeah, super important, of course. But why?

Yeah, because I feel like in many problems that, you know, where people are looking at data to get insight or any of these sorts of things, that if we knew exactly the question to ask and we had all the necessary data, we would just write an algorithm to do it. But I feel like that's rarely, if ever, where we're at. And I feel like that that's why data science has become such a big deal as you need people that can go in and explore and know how to statistically ask questions and provide variations on results and things like that. I think this is exactly where visualization is so important, and I don't think it's at the expense of statistics, but I think it's along with and this idea that if you don't exactly know what you're looking, looking for, that this is where visual exploration is just so important and. Yeah, so I just feel like it's hard to do data science without it.

Yeah, I'm not sure that's the public image that people have of data science, because I think most people equate data science to machine learning and with the problem of building predictive models, you know.

But, you know, even these, these biology biologists that I would work with, they would oftentimes just complain about these black boxes that the more computational people in their labs would give them, and they'd say, oh, I put my data into this one black box and I get this answer, and I put it into a different black box and I get a different answer. I don't know what that means. And so I think that it's not that we shouldn't have these algorithms, but this idea of how can we open that up somewhat, how do we help people better understand so that they can interpret these results? But I think there's also, you know, in talking with our machine learning faculty and stuff here, too, there's so much space for humans in the loop as well, you know, to these active learning types of models, where I think visualization is kind of a natural way to give people information to help them make better decisions, to help improve the models and so on. So I think there's just so many places where having that visual feedback is really useful.

Yeah, I agree.

And, I mean, even if you end up with a really nice black box, I mean, somehow you need to get to constructing that box. And, you know, like, for any predictive model, there's a long process of actually figuring out what is predictable in this context and how and what is the right. Sometimes you don't see that leading up to that black box that is later deployed successfully.

Yep. And then there's always the classic, you know, I think everyone, everyone who's done visualization design has an experience where someone loads in their data into a visualization tool for the first time. And what do they see? They see errors in their data. So even just from that low level spot checking, visualization's really good at showing you the things you weren't expecting to see.

The ugly truth.

Yeah, the ugly truth, exactly.

Yeah. And I think one interesting space where I see a lot of value in visualization is understanding what you actually feed your algorithm with. Because that's a huge problem in data science. Understanding, what do you exactly give as an input and whether it's valuable and whether it has a signal of some sort. It's really, really hard.

Hey, Enrico, I think there's some kids.

Yeah, it's my song.

Your boss is calling.

Yeah, that's my son.

I cannot open the door.

Otherwise, it's gonna be even worse than that.

I just got this great kids book called the Boss Baby. You should look into it.

Yeah, I need three of those. Is it only one volume or the old encyclopedia?

Unfortunately, it's just one volume. You could write.

I could write one.

The Bible. But, I mean, it's good. We all agree data visual scientists like breaking news. Data visual scientists agree that data science is obsolete. So that's fine.

Close that.

Case closed. No, but it's been super great talking to you, Miriah. That was super fascinating.

Guys. This has been super fun.

And keep us updated on the poetry thing. I'm really.

Oh, yeah, we want to see the poetry stuff. Absolutely.

Okay. I'll send it to you as soon as we have something to say.

Cool.

Yeah. Thanks for coming on the show.

Awesome. Thanks, guys.

Thanks so much.

Bye bye.

Bye.

Bye bye.

Data stories is supported by Tableau software, helping people see and understand their data. Get answers from interactive dashboards wherever you go for your free trial, visit Tableau software@Tableau.com. Datastories. This is T A b l dash, dash, dash dash eau.com Datastories.