Narrative Visualization Research w/ Jessica Hullman

Hey, everyone. Data stories number 40. Hi, Enrico. How you doing?

Hey, Moritz. I'm doing great.

Yeah.

Summer.

How was your summer? Did you have some vacations?

Summer was awesome. We spent some time in Florida doing, you know what? Nothing all the day. Just. Just going. Yeah. From our apartment to the beach and from the beach to our apartment. Nothing really special. Just in Italian, we say dolce far niente, which means doing nothing. Sweet. Doing nothing.

Sweet. Doing nothing. Yeah.

Yeah. How about you?

Yeah, same. I had some vacations, too. We went to Bavaria. So we stayed with my parents and Sina's mother, so they took care of the kids and we could, like, do our thing, which is, which is also good form of vacations. One week of working a bit on the house and planning a terrace, things like that. So cool. And now I'm back in the office. Had a full week of calls. I'm a bit tired of talking, actually.

Don't tell me. It's been crazy for me since when.

I'm back, there's so many things where I said, yeah, we'll totally do that in September. And now I'm totally in September. Have to do all these things and. Yeah, yeah.

Crazy.

I like work, as you know, so I should complain.

Yeah, same here. Should we start?

Absolutely.

So we have another special guest. Special, special guest today. It's my pleasure to have here Jessica Hullman from. Jessica, right now, you are at University of Berkeley, right?

Yes, correct.

California.

Berkeley.

University California Berkeley. So Jessica is a researcher in information visualization, and we invited her because we, I think what is really interesting about her research is that she's been one of the first persons in our community who's been publishing, or started publishing research about how to tell stories with data through visualization. And she's been publishing a number of very interesting papers on the topic and some of them even controversial. So I think it's going to be. We have lots of interesting things to talk about during the show. So, Jessica, you want to briefly introduce yourself, say, what's your background, what you do and. Yeah, this kind of stuff.

University California Berkeley. So Jessica is a researcher in information visualization, and we invited her because we, I think what is really interesting about her research is that she's been one of the first persons in our community who's been publishing, or started publishing research about how to tell stories with data through visualization. And she's been publishing a number of very interesting papers on the topic and some of them even controversial. So I think it's going to be. We have lots of interesting things to talk about during the show. So, Jessica, you want to briefly introduce yourself, say, what's your background, what you do and. Yeah, this kind of stuff.

Sure. Yeah, I'll try and keep it brief.

Yeah.

So I'm currently a postdoc, like you said, at University of California, Berkeley. I'm supported here by Tableau software. They're funding my fellowship. I am working with Maneesh Agrawala, who is in computer science here in the visual computing lab. And so we're doing basically infovis HCI ish research. So I, prior to this, just recently, at the end of last year, I got my PhD from the University of Michigan. I was at the School of Information there, and I worked with Eitana Dar, and again, I was doing information visualization, mostly some HCI work. Prior to that, I have a master's in information retrieval and analysis, also from the University of Michigan. So I was there in total over six years doing those two degrees. Before that, it gets more interesting, perhaps. I actually have an MFA that I did right before I did the master's degree, and that was actually in experimental writing and poetics, so we like that. And my undergrad as well is kind of more of a humanities focus, is actually comparative studies and religion. But all throughout sort of college and even in high school, I was always kind of both into science, but also into art, especially into writing. And so I kind of wrestled with both of those things throughout undergrad was like a pre med for a while, and then got this degree in comparative studies. But after that, I still wanted to sort of fulfill this dream of getting an MFA to see if I could become, like a writer. So for that, I was in a small buddhist university that has sort of a well known kind of experimental writing program in Boulder, Colorado. So I did the MFA there, and it was great. Although I realized, I think somewhere along the line, that I just missed sort of the science and the analysis too much. Focusing only on sort of, you know, creating something that's, you know, totally artistic was just a little too subjective. So I did this sort of rapid 180 after that point that started kind of pushing me towards visualization. But I initially, when I left that school to go get my master's degree in information retrieval analysis, my original goal was actually to do NLP. So natural language processing, because I was really into writing. I was really into sort of critical analysis of text also. So I was interested in, like, whether we could create software to tell kind of things, like who wrote certain texts. So sort of analyze a whole corpus from different authors and do kind of discourse analysis. So I went to get that degree, the masters, in order to see whether this would work for me. But it was during that degree that I kind of actually found that visualization just felt more right. So I found that NLP was still. I mean, it's still very confined within these limits of linguistics and then sort of statistics and probability. And I felt like when I found visualization, I took a class and just in information visualization and felt like there was a lot more room to sort of think about how people think in general and to potentially draw on other influences for more sort of outside disciplines. So I don't know, it just felt right. So I did that degree and then went right into the PhD.

So that's such an interesting background. Yeah, quite a journey. That's so interesting. So is there anything that comes from your training in poetics that you still use in your.

So that's such an interesting background. Yeah, quite a journey. That's so interesting. So is there anything that comes from your training in poetics that you still use in your.

So, actually, I did some kind of attitude.

I don't know.

Yeah, I mean, I think there's a few things. It's sort of vague, I would say. It's not like a direct influence. I mean, I was studying writing, so. And that included critical writing. So I did. I actually did a lot of critical writing on kind of aesthetic theory. I was working with an art historian. I got kind of lucky. This sort of well known art historian was at the school where I was getting my MFA, and I was able to sort of focus a lot on art history. And I actually got into sort of, like, analysis of narrative through imagery while I was doing that. So I think that definitely sort of set the seed in me where I was really interested in narratives that were conveyed visually. But I would say, I mean, more in general, I think getting an MFA, you kind of learn how to sort of tap into your creativity when you need it, kind of. And I think before I did that, like, in undergrad, I hadn't really experienced that sort of learning how to manage and, you know, use your creativity to your best, to the best effect. Like, when you need it, you can pull out some interesting idea and refine it. And so that helped. And then, I mean, more generally, just the critical writing, the practice, I got good at writing quickly. So when I started to write conference papers, it was a little bit of a challenge because scientific writing is very different, even from, like, critical, you know, kind of aesthetic theory. But I think it still helped to just have so much practice. So.

Sure, sure.

So you heard of. I think there is a group that is called something like visual poetics or visualization poetics. Are you aware of that, Moritz? I think Jen Low is working on that. I think it's a group in New York.

Okay, could be.

Yeah, yeah, yeah. Anyway, yeah, we need to ask.

So we have to invite Jen.

Yeah. By the way, we have to invite her sometimes. So let's switch a little bit to what I was saying at the beginning.

School of poetic computation.

School of poetic computation, yeah, that's Zach Lieberman.

He did a school, and Jen was teaching there. I think that's.

Yeah, that's really cool.

So what do they teach them? Is it. Is it more art driven or is it more science driven?

I think it's more art driven, I guess.

Yeah. It's like an experimental approach to computing and use computing in a poetic sense sense and make like computer experiments that touch people or, you know, play with emotions and.

Interesting. Yeah.

Have you ever seen that? How is it called something, something like Google. Google poetics. Google. Yeah, I think it's along these lines.

Yeah, no, that's totally cool. Yeah, I liked that project.

Yeah.

But I think there's a lot to discover in that space. And also the language angle to data visualization is one that I'm really interested in. And I think from many ends, everybody ends up there at some point to think about the language of visualization, right?

Yeah, absolutely. So, Jessica, how did you start doing research in visualization? How did it start?

Yeah, absolutely. So, Jessica, how did you start doing research in visualization? How did it start?

I mean, I think all throughout school, like undergrad as well, I just like research. Like, I like finding a problem and then working on it for a while and like really thinking about it. So I guess I just, when I was getting the masters at University of Michigan, I don't know, I came in wanting to do research and I didn't do a ton of it. As a master's student, I started to write papers. I actually, that's when I started to learn about like the Infovis conference and stuff. I think I tried to write a paper as a master's student. Didn't get into infovis. It was, it was kind of about storytelling. It was actually, I think it involved like, Arnheim's work. I don't know if, you know, I forget his first name, Rudolf Arnheim. It's, he, he talks a little bit about like, you know, how we, we see these kind of implied dynamics when we look at visual stuff. Like we almost see it moving depending on how the strokes are. So, yeah, so I remember doing that, and that was with someone who initially was my PhD advisor when I started. And so, I don't know, it just felt natural, I guess. I'd always liked sort of independent projects.

Thinking, I think what is really interesting. So when I look at, at your publications yesterday, I was preparing for the show and I thought, well, I think the visualization community, I mean, the research visualization community has been focusing for a very long while, for a very long time, more on the perceptual side of visualization. And even the kind of few guidelines or theories we have out there are mostly based on the low level perception of visual variables and stuff like that. But then when I look at your work, it looks more like you started very early on looking at cognitive effects. Right. Rather than starting from the perceptual side of things. I don't know if you agree with me that.

Yeah, I think that's a tricky question. So I agree that there's been a lot of focus on perception and sort of the history of visualization research. I think cognitive, or cognition is a tricky word because what does that really mean? So, I mean, I think a lot of my work, I'm interested in where sort of person perception and how we process information just sort of automatically as it comes in, either visually or the order in which it comes in. I'm interested in where that begins to shape cognition. Or, I mean cognition, I guess, meaning sort of the actual interpretations we draw. So there's sort of these, I guess. Yeah. Cognition is hard to define. If you mean, like, sort of conscious thinking about things, I think it's potentially that's less where we're seeing this expansion in visualization, and it's more sort of where these kind of more automatic influences begin to shape interpretation. So it's like the meeting point in my mind. But I mean, those words can mean a lot of things. So, I mean, what do you think cognition or cognitive is? When you ask that question?

It's hard to tell, but I think. I don't know. I think most of your work is, if I interpret it correctly, is more about how people interpret or get a message out of visualization. Right?

Yeah, yeah.

And at least for me, when I saw for the first time the kind of papers that you were publishing at the visconference, I was really struck by this kind of new angle of visualization research.

Yeah.

And I don't know, maybe just was natural for you, but when I saw this kind of papers, I was actually surprised, because I don't think that we had anything similar in the past ten or 15 years, at least in our small community.

Huh. So, yeah, I could see, I mean, I think what I'm interested in, and maybe this unites a lot of my work, is sort of the mechanisms by which we arrive at an interpretation. So, like, what's happening in our head that makes us believe one way or another when we see a visualization. And so I would say some of the work I did on the visualization rhetoric or storytelling was about how these strategies that designers use are tailored to make our brain work in certain ways to arrive at interpretations. And some of the other stuff I've done as well, the visual difficulties, I think, is about how we think about stuff and how certain ways of thinking about things that we can't always control will have an effect. On sort of what we learn. So in that case, it was like active processing. So if you can present information in a way that people have no choice but to really engage with it and think about it, they might learn more from the visualization. The storytelling stuff is more sort of use of different presentation strategies, or they call them framing strategies, in like, political science and communication. But it's basically you're presenting the information in a way that attributes related to how it comes into the perceiver's head are going to lead them to a certain interpretation. So the order effects is a really simple example of that. If you present things in a certain order, we as humans have this tendency to prioritize certain pieces of information based on the order. So we often either prefer the first things we heard or the last things we heard. It depends on the context. But so the storytelling work, I felt, I guess, when I wrote that paper, I didn't expect it to really get published at all, but I just felt like it was sort of this glaringly obvious thing, that visualizations were not just about this sort of very careful, deliberative thought where we're looking at the data and just sort of analyzing it. They're also about somebody trying to convey a message and the way that they convey that message interacting with these different tendencies or biases we have and all of that sort of feeding into the interpretation.

Is there a connection to speech act theory? In a sense that, I don't know.

Too much about speech act theory, so maybe you can tell me that.

But I was just reminded of that because, I mean, linguistics has all these different fields. So you can look at the syntax, like how you can construct sentences, semantics, what words mean, what words construction means. But one part is also pragmatics, like how you use language, like what's the practical meaning implied by, you know, you can utter the same sentence in two different contexts and semantically it means the same. But what you imply by saying it.

Or by asking, yeah, no, I definitely.

Think there is, you know, that makes it an act in the world and that makes it something, an intent and something that is being interpreted.

I think there's a lot of conventions or expectations that we don't think about. So when people see a certain type of visualization, how they've seen that visualization used in the past is gonna impact perhaps what kind of message they think they're gonna get. I think actually Barbara Tversky and her colleague Zachs, I can't remember his first name, did an interesting study that I think is a nice example of convention or expectations. It's the one on bars and line graphs. So if you show someone a line graph, they think in terms of trends, regardless of whether the data is actually trend based. If you show them a bar graph, they tend to think in terms of these discrete groups. And so you can show people discrete information, but if it's in a line graph, they'll want to interpret it as a trend. Yeah. I think that kind of stuff is a big sort of overlooked thing in visualization. I guess I'm just really interested in sort of these expectations that are harder to always see but that are having this big influence.

Yeah. And I mean, designers work a lot with that, but they have a hard time formulating it. You know, it's more like implicit knowledge and a certain gut feeling you develop. But there's no, no good formal theory, I think, of how that works. It's great somebody's working on that.

I'm really intrigued by how designers do like, you know, do what they do, like, how they think about it. I think a lot of my work, I've, especially lately, I've done sort of more systemy stuff where I'm trying to figure out what designers like, what kind of rules they're using in the first place. But it can be hard because it's, I don't know, I need to perhaps do more studies where I actually interview designers, but it's hard to get at what they're actually doing. I don't know that they could articulate.

It, but, yeah, this is what I often think about the fact that now we are at a stage of visualization development where finally we have so many interesting people out there who are doing visualization every day in practice. Right. Lots of designers. And I'm surprised that there is not a lot of research that takes advantage of this wealth of knowledge. Right. Just going there and trying to, I don't know, kind of like this kind of even ethnographic studies where you sit together with a bunch of designers and try to observe what they are doing and trying to understand, I don't know, try to see if there is anything interesting there. Yeah. I mean, even just a few years back, it was not possible because visualization was not as popular as it is now. Right. But right now, I think there are so many interesting people out there and, yeah, I haven't seen anything like that.

We don't think perhaps enough about process, like how.

Absolutely. Absolutely.

With which a visualization is created. I think a lot of this sort of early advice from Tufte and others was that there's only really one way to show the data correctly, and so you just find that. But I think there's actually more. There's a lot of decisions, I think, is my sense, although I'm not really a designer, so it's.

Yeah, I think. Yeah. You're touching upon a very interesting point, because if I look at the way I develop my sense of what is a good visualization, through the years, I have been trained. I've been trained with this kind of tufts worldview. Right. And in a way, through the last few years, I've been modifying this view to some extent, and. And it's been a really interesting process, in a way, even painful.

I mean, talking of Stephen Few, we could talk about the visual difficulties paper because I think it's a really. It was recognized by Stephen Few as the result of ill conceived and poorly conducted research. So that's quite an honor, I think.

Is it? I'm not sure. No.

I think everybody should have been bashed by Stephen fur once in their career. I think it's an achievement, probably.

If you don't have haters. I think my advisor told me this. If nobody hates what you're doing, you're not doing anything interesting.

Right. Right. So can you tell us a bit about the paper?

Right. Right. So can you tell us a bit about the paper?

Yeah, so that paper. Well, it. I was working with a cognitive psychologist who I worked with a lot during my PhD, Preeti Shah. And so I would say that she initially got me into thinking about sort of these different views of how we think with visualizations, and even outside of visualization, how we think in general and what's conducive to thinking when we're trying to learn something. And so I think, again, I did those two papers the same year, the visualization rhetoric and this other one. But I think they both kind of felt like, well, this stuff needs to be said, you know, like, there's this whole other viewpoint, and we shouldn't just ignore it in favor of sort of this, like, very minimalistic type of design. And so I think a lot of the evidence from other fields, like educational psych, for instance, that shows that when people sort of think more actively about information, make themselves predict things before they look at the answer, they actually learn it better, was so persuasive to me that it just seemed like something that might apply to visualization. And so I started looking at visualization and seeing some examples. So examples with animation are an example where animation is not always as useful as we think. Although it's less work for us, we don't always learn things or remember things as well. And so, I don't know, I came up with some examples working with preeti that kind of reflect cases where you want someone to think a little harder because the sort of superficial interpretation is one less likely to be remembered because they haven't processed it as sort of actively or thought about it as deeply. But it's also potentially misleading in some cases. So, yeah, I mean, I think the paper was intended to be a bit of a polemic to sort of, you know, raise this other view. But I think there's really a balance between sort of knowing when to make something easy to perceive a given pattern and knowing when to make someone think more deeply. I think it also depends on things like individual differences. And so, I mean, we try to sort of go over these different factors in the paper. So who the person is, how much they like to be challenged, are going to be important in determining how challenging to make your graph. But yeah, I think a lot of it, difficulties, I'm not sure is the right word because I think people reacted sort of strongly to that word. Perhaps just sort of engaging or sort of active processing with visualizations, I think there's a clear link between active processing and memory. So it sort of strengthens our, the connections in our memory when we think about data sort of for longer or more deliberately. So it's kind of strategies to get at that can be useful. But yeah, yeah, I think it's. There's an interesting line of research.

So you're not necessarily about more complexity, but more about you worked on acquiring insights for yourself rather than a lot of assuming like spoon fed information. Is it like that?

Yeah, yeah, I think that the sort of driving point of that argument is based on these examples where there's a little more work and sometimes, sometimes it's sort of conscious work. So sometimes it's because people, they don't understand the visualization at first. Perhaps the design is some sort of fancy tree visualization, and it's not immediately obvious what the point is. So in some cases, I think they might make an active decision to process it further, but in other cases, it doesn't even have to be something that they're conscious of. So we cite a lot of research on difficult to read fonts because it's a clear case where it's been shown that it's. People think they're doing something more difficult. They don't necessarily try to think about it more than they would or try to think about it deeply, but the brain, because it sees that difficult to process, like visual form actually flips into sort of a different mode. There's, I don't know, the system one versus system two thinking where we have, you know, two different ways to process information. I don't know that it's actually that clear. There's these two distinct systems. But I think there's, you know, some visualizations tend to present information in a way that's designed to give us this immediate sort of perception. And I think I was just in the paper, we're just trying to say that that's not the only way, so.

Yeah, as well. Right. Isn't that tied to memorability? Maybe?

Yeah, it is.

I think to have a unique experience and you can tie information to your experience.

I think the memorability stuff is really based on kind of like depth of processing. So if you get someone to think about it more deeply and more actively where they actually, they want to learn something, like they realize that there's a gap in their understanding and they need to fill that gap. That typically, I think, makes the memory stronger. The encoded or the process by which you're encoding the information is more developed and somehow the memory sticks. I'm definitely not an expert in memory, but I do know there's a tie between sort of depth of processing and then memory, both long term memory but also short term memory. And so, I mean, I think things have been interesting in visualization where other people have also been looking at that memory piece. There's been some interesting work, like Scott Bateman's work back at CHI's was kind of cool. But. Yeah, it's tricky. I think memory is a complicated thing, but I think there's also probably individual stuff there, like how much do you care about the data? If you really care about the content, you might remember it.

Surprise is affected too. I mean, it's.

Yeah. Whether you expected it or not.

And I guess there are lots of individual differences as well. Right. I think this is another aspect that is not very much studied so far in visualization, but I think it's huge.

Yeah. So they've studied it more in, I think, cognitive psych, like graph comprehension. But actually, I read your persuasion paper, the one that you're involved with, and I really liked that you guys, you talk about sort of the initial attitudes, I think.

Oh, yeah, yeah, yeah, absolutely.

That people had and how that's an important factor.

Yeah, yeah.

So what made you look at that so.

Sorry, say it again.

What made you look at that in the first place? Did you just have a hunch that.

Oh, yeah, it's just that it's, I think it's an interesting story. I think it starts three or four years back. I was invited for a talk to development agency in Canada in Ottawa. I give this talk, and in the middle of the. No, at the end of this talk, one guy raises hands and says, can you persuade somebody with a chart? And I was like, oh, man, I don't know how to answer to this question. And I was struck by, I mean, I don't know why I never thought about this question before. And I didn't really have even a clue how to answer to this question. And he said, look, that's the most interesting kind of question, the most interesting kind of information for me because I'm confronted every day, almost every day with this problem. I have a boss. I have to convince him that what I'm presenting is important and real and he has to take actions based on what I'm presenting. Right. And then I came back home and was like, oh my God, I should do something. Right? So. And, yeah, but then for several circumstances, I didn't have time or resources for doing research in this area. So I had it in the back of my mind for a very long time. And, yeah, last year I decided that finally had to do it. Yeah.

So you guys, you did a study, right?

Yeah, we did a study, but it's. Yeah, man, it's been really, really hard. I don't want to do it again. It's been crazy.

Try to summarize it. I think it's kind of interesting.

Oh, yeah. Well, that's not my interview, but you get three minutes.

It's fine.

Yeah, let's get three minutes. So I think we just, I think our research was mainly about, I think we really started with the idea of, can we even study persuasion with charts or not? Right. Because it's not clear how to set up a study to study this aspect. It's really, really hard. Much, much harder than I expected. So basically what we did is we created several stories, and then we present the stories either with charts or tables, and then we see how the attitude of the participants change, and then we compare how much or to what direction it changes with charts and how much it changes with tables. And after several iterations, though, right? Exactly. Same story. Exactly. Same data. Yeah. We had to control for a lot of different things. I wanted to be tedious, but. So we had to control. We actually had to run a lot of test studies, pilot studies, to understand whether we were doing things right or not. And what is interesting is that we found that it looks like that the effect depends on the initial attitude of the participants. So if the participant is in favor of the argument that you are proposing, charts can make people more persuaded. Okay, but this is what our data says, then of course, you have to double check. Right? But if the participant is against the argument that you are proposing, it looks like that the participant is more persuaded when he or she is persuaded by tables than charts. And we don't have very good explanations.

Yeah, let's get three minutes. So I think we just, I think our research was mainly about, I think we really started with the idea of, can we even study persuasion with charts or not? Right. Because it's not clear how to set up a study to study this aspect. It's really, really hard. Much, much harder than I expected. So basically what we did is we created several stories, and then we present the stories either with charts or tables, and then we see how the attitude of the participants change, and then we compare how much or to what direction it changes with charts and how much it changes with tables. And after several iterations, though, right? Exactly. Same story. Exactly. Same data. Yeah. We had to control for a lot of different things. I wanted to be tedious, but. So we had to control. We actually had to run a lot of test studies, pilot studies, to understand whether we were doing things right or not. And what is interesting is that we found that it looks like that the effect depends on the initial attitude of the participants. So if the participant is in favor of the argument that you are proposing, charts can make people more persuaded. Okay, but this is what our data says, then of course, you have to double check. Right? But if the participant is against the argument that you are proposing, it looks like that the participant is more persuaded when he or she is persuaded by tables than charts. And we don't have very good explanations.

Because numbers don't lie.

Because numbers don't lie. And then we also have some kind of analysis of a series of open ended questions that we asked. So I think that's an interesting part that is more qualitative and describes why people do or do not change their attitude, and they have all sorts of explanations. And some people are quite clever, actually. So this is one thing that it's really important to say. I think that we believe we tend to consider most people quite not very knowledgeable and not able to distinguish, to discern between a good or bad chart or good statistics, good numbers. But I think it's not as evident as we, as we think it is, because we, in our study, we got a lot of people who actually wrote a lot of clever sentences explaining precisely why our argument wasn't actually a good argument for our data wasn't enough, or our statistics were not supporting our arguments. So that's really interesting.

I think I've heard something before about if people, if someone doesn't agree with an issue or an argument, that presenting it actually makes them agree even less, like it makes them present, or that makes them sort of reinforce this counter argument in their head.

Yeah, yeah, yeah.

So that's really interesting.

No, but if you look into the persuasion literature, it's crazy. I mean, it's a kind of worm.

So there's. So you were going after kind of persuasion, like an actual. Is it like sort of subjective topics that you're going after or subjective arguments or.

Yeah, yeah, yeah, absolutely.

Yeah. I think there's been a little work in just how does, like, when the data is correct and people are less likely to argue with that. Like, how does showing it in a graph change, you know, versus change people's use of the data over showing it in a table? And I think there, they found some interesting stuff. It's kind of like things like graphs do make data more salient. There's more differences that pop out at us based on the size and the shape of the visual stuff that we just don't see in words or tables. But that I think one of the studies actually showed, too, though, that people, they don't always remember the specific numbers as well. When they see it in a graph, they remember. It's more of an effective response. So maybe you riled people up even more by showing them persuasive visualizations that had some argument they disagreed with.

Yeah. It's a really interesting question, and it's complex, unfortunately. It's very complex.

Yeah.

And you have to rely on people self reporting that they changed their mind. And, I mean, that's the big man. That's such a. Yeah.

Yeah. It is shaky.

It's hard.

Did you do this on mechanical Turk, this study?

But what is, I think what convinced us that this was, let's say, publishable, at least, is that we repeated exactly the same thing with three different stories without reusing the same participants, and the trends are very similar.

That's good.

So that was really surprising, actually.

Yeah, I do. I like what you said about sometimes we think people don't really think very hard about charts or they don't understand what we're saying.

I think we underestimate people in general.

Definitely.

We underestimate the layman.

Definitely. Yeah, I think that's actually. I think even Daniel Kahneman, who did a lot of the cognitive biases, stuff like a lot of that literature, is showing people are flawed and how they reason about information. They're not sort of perfect Bayesians or whatever. But he wrote a book, the thinking Fast and slow book. I don't know if you know that one, but I think he says somewhere in there that a lot of his later work has actually been trying to convince people that even though we have these natural tendencies and they're not always perfect, they're actually kind of sophisticated, like all these heuristics that we use and our ability to sort of, like, actually use them in a way that works for us and, you know, helps us process information so that we can at least make the decisions we need is something that we should not, you know, that we should be appreciative of. It's. There's some sophisticated stuff going on, kind of. So. But I feel that way a lot. I think when I do a lot of research where I'm trying to, like, understand how people who are not necessarily statisticians use charts. And I've been doing a lot of studies on Mechanical Turk, I've always sort of used that as kind of a proxy, which, I mean, it definitely has its downfalls. And, yeah, it's kind of tricky at times, but I think it's been really useful at sort of convincing me as well that, you know, even people who haven't taken a lot of statistics or have used graphs, much like they're, they're capable of reasoning in a very rational way. And we do underestimate, I think, just sort of the general population sometimes.

Yeah. Yeah. So you've been doing some work on understanding how people look at samples or uncertainties. Can you tell us a little bit about that? I think it's really interesting.

Yeah. Yeah. So you've been doing some work on understanding how people look at samples or uncertainties. Can you tell us a little bit about that? I think it's really interesting.

It's something I've been interested in for a while. It's work I started with my advisor when I was getting my PhD, Eitana Dar, and we were talking about this idea that if you could show people sort of actual samples of what could happen if you repeated this process that produced your data, maybe then they would understand the potential for variance or the potential that a pattern that they see in the original sample doesn't necessarily persist if you repeat the data. So I guess, I mean, it's kind of based on this view of uncertainty as sort of a need to understand that other outcomes are possible. So we could argue about that definition of uncertainty or not, but I think it's reasonable that we're trying to show people the potential for variance. And so we started thinking about this idea of just show them directly the hypothetical samples. So there's ways you can resample your data to produce additional samples that sort of represent hypothetical outcomes. So there's like bootstrapping and statistics and then other forms of resampling depending on what kind of data you have, that can help you produce sort of an equivalent data set that might have resulted from the same process that produced your original sample. So it's something that you could apply to any sort of data. And where it's helpful, I think, is that a lot of the sort of conventional ways of visualizing uncertainty are, for the one part, kind of framed for particular visualizations. So error bars are something we see a lot, but they're really made for bar charts. Sometimes you can use them on, like, scatter plots, but, or line charts, but it's. They have sort of a small number of formats. And so there's this problem, like, how do we show uncertainty when we have something much more complicated, like a network diagram or, you know, some complicated maps? And so one nice thing about showing it as these poten or these hypothetical samples where you show someone a set of them, maybe you show them small multiples or an animation, is that you can actually do it for any type of chart, but the other big problem, I think, with some of the conventional uncertainty representations is that for things like error bars that are showing a confidence interval, you kind of have to understand the statistics behind it. So to understand an error bar and to compare two different error bars, you have to know what that interval actually means. And what does it mean when your error bars overlap. And there's actually some studies I found that people even who are experts who are using error bars all the time, they don't quite understand, like, the concept of statistically different and how you get that from error bars. And so I think what we found, it's been really tricky doing studies with these comparative sample plots. And what we're doing, like I said, is like small multiples. Like we either create a whole bunch of samples, like hypothetical samples, and we show you like 16 or 20 of them, or we show you an animation that's actually, it's playing through all these hypothetical samples. So it's like you could have a bar chart, and the bar charts actually sort of animated from one frame to the next, you're seeing a new sample. But it's been sort of tricky in that we've been showing people these things. On the one hand, they're a little bit harder to process compared to, like, a bar chart with two bars and error bars on each bar, because you're getting more information, you're getting all of these samples. But on the other hand, the fact that they're less abstract, that you don't have to, like, understand the error bar thing, appears to work. Like, we've found that they're actually more effective for helping people estimate the probability that a pattern that they see is going to repeat compared to error bars. So it's giving you directly all the information you need to calculate. If I had more samples, how many times would I see that a is greater than b? So for that assessing the probability of patterns, it appears that they really work. So I don't know, I find it exciting, the whole idea of it, just because you can apply it in many cases. But there are. I mean, it's. It's tricky because it's a little more difficult for people. I think it's, you know, there's like this initial, like, oh, it's. The chart is moving, you know, for these animations. Like, yeah, I can't do this, or.

Like, which one is it now? You know, you show me which one is. But did you know New York Times did something like that? Like, they had an article on how the soccer World cup, how the groups are drawn. So there's a certain mechanism how the groups are formed, like based on the strength of the teams. And they made an argument that the mechanism is sort of flawed and leads to very skewed. So you often end up with a very strong group and a few weaker ones. And they showed first the histograms and the distributions and the model aspect of it, but then there was also a button, and you could sort of simulate a draw based on the old scheme or the one they proposed. And you would see immediately. Yeah, hold on. The right hand side always looks better, you know, so you get this gut feeling of, yeah, it seems to work. And then you go into the math and that's, I thought it was really nicely done. And you could just press, reshuffle or redraw and it would simulate a new run.

Yeah, I think it kind of goes back to that thing on process. Like, we don't always think about the process of creating visualizations and like the potential for error or noise, like when you're actually collecting the data or when you're graphing the data, that the fact that there's a little bit of noise or variance. And so what we actually see in the end isn't necessarily the absolute truth. I don't think people think about that when they see a visualization. I think they think this is fact or whatever. And so anything you can do to kind of get people to think, oh, other things could happen is potentially.

That's so important. And so going back to the New York Times example, I think I really, really loved that piece because when I saw it, I was really, I mean, what really surprises me about this New York Times piece is that they had the gut to do something so nerdy. Yes, so nerdy. Right. I mean, if you look at the details, it's, I mean, the statistics is not, it's not obvious. Right. And I would be totally scared to expose these kind of things to a large, large audience. And they did it. And they did it in a very beautiful way. I mean, it's, it's really, really, really nice. I think this is another interesting trend that I think it's where I think visualization is at all. I think that there is a, I have a hunch that there is a much larger segment of the population right now that are exposed to numbers and statistics in general. And I guess that this is actually leading to people being much less afraid. Right. Of numbers and stats. Yeah.

We need to work on, like educating people to work with models and understanding predictions. I think, you know, when you talk about climate debates or economy. I mean, anything. You know, we're always surrounded with these predictions. And if you're not a scientist, it's very hard. Even if you're scientist, let's face it, it's very hard to make sense out of that.

Yeah, no, I totally agree. There's always assumptions built into models and we so rarely see those. Even like, I was thinking about things like averages. We see averages all the time. And I think people probably have a good idea of the need to average rather than take a single example. But I wonder sometimes, is it always clear how you can get a very different average depending on, you know, what your data actually looks like? You could have outliers that are totally skewing something like, I mean, I think people are capable of thinking through these things, but we don't always know enough about how to sort of cue them to start thinking in the first place. So then you're.

That an average of averages is usually not a good idea. That's also something people are usually not familiar with until they ran into that issue. And it sounds plausible, but yeah, yeah.

That an average of averages is usually not a good idea. That's also something people are usually not familiar with until they ran into that issue. And it sounds plausible, but yeah, yeah.

Maybe visualizations need to do more teaching, like along the way or something. I don't know. I'm gonna have to check out this New York Times one because it sounds really cool.

I think this also nicely introduces the problem of visualization literacy, or even statistical literacy in general. I think that's a really, really interesting aspect. And again, that's something you've also been.

Doing a little work on, right?

Yeah, but we shouldn't talk about my work.

You're always pushing for opportunities to present your work.

Well, I'm prompting.

Apropos your work. Well, I have been working on. Can I ask something about Jessica? You have an upcoming paper on content, context and critique. Commenting on a data visualization blog, that's.

Actually, that kind of ties with the visualization literacy thing. So that's actually just a sort of qualitative study that I did with Nick Diacopolis and also Lehe, momene, Ruchi and Eitanadar. And so we were interested in this economist data blog. So the Economist, the magazine, has a website and they have this particular blog that's called daily charts or I think, graphic detail. They've changed the name, but it's every day. They are not every day. Every working day they present a visualization, and then they get tons of comments on the visualization. Visualization is always something related to economics. And so we started with the blog cult again.

Can you say it again?

What's the name of the blog I'm trying to think of, it's called graphic detail. They changed the name, yeah, from daily chart to graphic detail. But one thing that we noticed, me and Nick in particular, were kind of really interested in the discussions going on there because it a lot of times you think about visualization and commenting on visualization, and it hasn't always been successful. So you get a lot of comments that are just like, oh, wow, this is really cool. On some visualizations, many eyes would be an example. That site, I think, was originally intended to get all these comments and you don't really see a ton. But this economist blog was interesting to us because it had so many comments and so many of them were really kind of astute. Like they'll go into the sort of details of a certain type of map projection and whether or not it was actually a good idea to use that. So it's really getting critical about the process. And so we did this qualitative study originally, just trying to get at sort of how does commenting on a visualization differ outside of these sort of research systems? Because a lot of the commenting platforms, like census or many eyes have been kind of coming out of research and they're sort of prototyping. And so it was really interesting because it convinced me that there's some very highly data literate people out there who may not even be working with data every day, but they like to reason about it. And when you put a graph in front of them, they want to go deeper and really think about how it was made and what sort of assumptions go into it. And so the paper is just kind of about that, that people like to critique decisions made in the process of creating a visualization. They like to focus on the content. But I guess the other thing that we found was that the context, so all of the sort of other pieces of related information that actually aren't shown in that data visualization, but that might be important to understanding it, are things that people are aware of and they're thinking about when they look at a chart. So sort of the whole backstory, all the causes, all these things that you don't actually see in the chart that might be important.

So how did you go about, did you sort of classify the comments into.

Different groups or sort of a grounded theory? Nick Dicopoulos is actually sort of good with his qualitative methods. Give him credit for sort of teaching.

Me how to invite Nick sometimes.

Yeah, totally. But, yeah, so I think it's sort of related to literacy in that I think I agree with what you just said, Enrico, that, you know, people are becoming more literate, especially with data. They're learning how to think about data, to critique data. But, yeah, I think there's still a long way to go.

Yeah. I mean, I don't have a way to demonstrate that, but I think the biggest difference is that now this stuff all in a sudden is cool. Right? And it used to be exactly the opposite. Right. Totally uncool.

Yeah.

So it's an interesting difference. I'm really curious to see how my kids are gonna see this kind of stuff when they grow up. Right. Maybe they would think that their dead do some cool stuff rather than boring. I don't know, weird.

There's like this nerd culture that's taking over.

Yeah, yeah, it's really weird.

But it's interesting stuff. I mean, like, it is. It is thinking about data. I think people, once they learn more about it, like, it's really interesting to think about statistics. Like, maybe not for everybody, but I think a lot of people can, like, sort of get into it.

Yeah. And what is really interesting for me is that it's very interdisciplinary. Right. There's no single angle that is going to work for everything. Right. So you have people coming from so many different areas, and you need all of them in order to do, to make progress. Right. And so you have computer scientists, you have designers, you have statisticians and so on, geographers, whatever, right. And that's really interesting.

So is that data science then, what you're describing? Would you say that.

Sorry.

Would you say that what data science is? I think people use that term a lot.

Well, of course. I mean, we could talk for hours about what data science is or is not. But I think for me, data science is more. I don't know. We don't want to go there.

Yeah, I've been asked that question. Yeah, it's a hard question.

Yeah, but I agree.

I mean, I think more people realize. I think it has to do about also about control, like this whole thing of program or be programmed. Like, I think more and more people realize you need some technical skills, just not to be, you know, to be an actor, actually, and take control of your environment and otherwise, it's very, like, empowering be part of somebody else's program and. Yeah, more and more people realize that, which is great.

Yeah, no, I think it's like, I learned to code pretty late. Like, I mean, I had done all these other things, but it's. It's definitely empowering. I think statistics are empowering. The more you can understand. Not that I understand a ton, but the more you understand, at least the closer you get to just feeling like, okay, I really know what's going on. So I think people sense that.

Yeah, I'm curious to hear how you learned how to code, because lots of our listeners, I mean, this is the typical question that we get for people who start doing this, and they ask, first of all, if they, in order to do this, you need to code. And I tend to say kind of, yes.

Yeah, I agree, sort of.

Right. I mean, of course, you.

For the most part, yeah, you can.

Use Tableau, you can use excel, you can use a lot of stuff, but if you. Code is much better.

Yeah, ultimately, I think you have to control it. Like, I mean, there's so many sort of rules built into any system that's going to allow you to avoid coding that, you know, like, unless you want to be subject to somebody else's constraints, you have to be able to create things yourself. That's the only way you're going to get the control. But, I mean, for me, learning how to code, and I'm not a great coder, was just. It started, like, I started trying to teach myself, but I would say as I was finishing this MFA, so it was, like, really weird. Sort of had my foot two worlds. But I would say I really started to get better just by throwing myself into this, you know, information retrieval, analysis, master's program and taking a bunch of classes, like, my first semester, like NLP, that just, like, I had no choice. I had to, like, learn really quick.

Right? Like vector space.

Yeah. And I'm not gonna say I understood it all, but I don't know. I mean, it just. I wanted the challenge, I guess. I'd always liked math and science, though. Like, that was always something that, like, I really liked, and I did well at. So it was. I don't know, it felt like sort of trying to refine that, you know? Like, I really liked statistics in undergrad. I remember loving the two statistics classes I took. And so I started when I was trying to code, just kind of trying to think about other things like that. You know, it's all like the. I guess, the left side of the brain, they say. I don't know if it is or not, but sort of just. I think all those things go together and you just. It takes a while, but you start gradually immersing yourself in. In that kind of thing. But, yeah, I don't really have any good advice for people learning how to.

Code, other than trying for a very long time. I think when we say learning how to code, this can mean so many different things. Right. I think that for a long time, people saw coding equated being able to code to being a good software engineer, which is totally different. Right. One thing is to be able to do some coding, and one thing is to be a solid software engineer. There is a huge difference between these two things. And guess what? In order to do some nice viz, not necessarily you have to be a very good software engineer, I guess.

Yeah, no, definitely.

If you are, it's better, but if you're not, you can still do a lot of good things.

No, definitely. I think it's more just kind of understanding how to think like a computer and stuff. That's probably the hardest part of coding. I mean, for me it was just, you know, you learn how to debug and stuff, and until you get that, you're kind of struggling. But I don't know. I mean, it's. It's possible, I think, for anybody.

Like, I've been skating by ten years and nobody noticed. But you, who I've been skating by for ten years now and nobody noticed.

Yeah, yeah, but. Yeah, but I don't know why. I mean, I get this question very often because people are scared. And I also think there has never been. It's never been so easy right now. There are so many resources out there and starting to go. There's never been so easy as it is now.

Yeah, definitely.

You can just grab a bunch of, I don't know, coursera courses.

Oh, yeah. That has changed a lot, too.

Yeah, it does change a lot.

Yeah.

So, Jessica, so what do you think is coming up next in this area? I mean, storytelling has exploded as one. I mean, people are crazy about visual storytelling, except me.

No, that goes back a bit. But we had a long debate, and I'm not.

Yeah, no, I don't think any of us are fond of the term.

Yeah.

A few episodes back, we had Alberto Cairo and Robert Kosara here.

Yeah, I think I listened to parts of that one. Yeah.

Yeah. But I think what is really interesting for me is, what do you think are the most interesting lines of research for the future? What? I mean, the best possible world in a few years. What would you like? What kind of research would you like to see there? Right.

Yeah, I think. Well, it's going to be very biased based on what I think is interesting. I think a lot of the stuff related to how these different kind of heuristics and cognitive biases or just shortcuts that we use. I think there's so much that we know from sort of the decision science literature about how they operate when we're presented with information that's not visual. But I think there's still a ton of work to do to figure out how do things change when you have a visualization? Do they change at all? So I think there's a lot of ways in which when we have to make a decision, we look for other information when we don't quite know how to make it. And I think there's questions about how visual information might be used to sort of make decisions about things like uncertainty without actually having the uncertainty information there. So I guess, I mean, one thing that I think going forward is just people. I think we'll begin to sort of explore these individual kind of shortcuts that people use when they're reasoning about data and try to kind of develop just an understanding of how to design visualizations in a way that accounts for those or expects those different shortcuts even more. I guess the stuff I did on order effects is kind of in that vein. So I think. So other stuff going forward, I mean, there's so much, I think there's a lot of cool tools that are being developed for creating storytelling visualizations, if you will.

Yeah, I think. Well, it's going to be very biased based on what I think is interesting. I think a lot of the stuff related to how these different kind of heuristics and cognitive biases or just shortcuts that we use. I think there's so much that we know from sort of the decision science literature about how they operate when we're presented with information that's not visual. But I think there's still a ton of work to do to figure out how do things change when you have a visualization? Do they change at all? So I think there's a lot of ways in which when we have to make a decision, we look for other information when we don't quite know how to make it. And I think there's questions about how visual information might be used to sort of make decisions about things like uncertainty without actually having the uncertainty information there. So I guess, I mean, one thing that I think going forward is just people. I think we'll begin to sort of explore these individual kind of shortcuts that people use when they're reasoning about data and try to kind of develop just an understanding of how to design visualizations in a way that accounts for those or expects those different shortcuts even more. I guess the stuff I did on order effects is kind of in that vein. So I think. So other stuff going forward, I mean, there's so much, I think there's a lot of cool tools that are being developed for creating storytelling visualizations, if you will.

So you've been creating some, right? We didn't.

Oh, I wasn't talking about mine, but we can't talk.

Yeah, but I think you developed a couple of.

My stuff is the stuff I've done in sort of automated visualization generation has really been about kind of looking at visualization and context. So the other information that often is presented with a visualization. And specifically, I've been looking at visualizations in news media and how often. And you have an article and a visualization and they sort of play off each other. The visualization helps you understand the article. The article helps you figure out where to look in the visualization. And so I've done a few systems that are really about sort of looking at this problem where we have so many sort of text articles, like news articles, and many of them are somehow related to data. They're about some disease that's some that's spreading across the US, et cetera. But there's not always a designer right there at hand to make a visualization. We've done some systems to try to figure out ways to take a news article and automatically find data that's appropriate and present the data in a way that's appropriate to the sort of comparisons that you would want to do between the data based on the text of the article. But when I brought up sort of systems for visual storytelling, I was also thinking of things like some of the work. I think Jeff Harris had a few systems like Lyra, that some of his colleagues have been working on that are kind of designed to help people create sort of visual stories more easily. I think there's probably still a lot to do there. Some of the stuff going on at Tableau, I think, is getting into sort of how to help people do things like annotate visualizations, use these sort of basic devices in a way that's just easier than it currently is. So I think that's potentially something we'll see more of. Yeah, there's so much.

My stuff is the stuff I've done in sort of automated visualization generation has really been about kind of looking at visualization and context. So the other information that often is presented with a visualization. And specifically, I've been looking at visualizations in news media and how often. And you have an article and a visualization and they sort of play off each other. The visualization helps you understand the article. The article helps you figure out where to look in the visualization. And so I've done a few systems that are really about sort of looking at this problem where we have so many sort of text articles, like news articles, and many of them are somehow related to data. They're about some disease that's some that's spreading across the US, et cetera. But there's not always a designer right there at hand to make a visualization. We've done some systems to try to figure out ways to take a news article and automatically find data that's appropriate and present the data in a way that's appropriate to the sort of comparisons that you would want to do between the data based on the text of the article. But when I brought up sort of systems for visual storytelling, I was also thinking of things like some of the work. I think Jeff Harris had a few systems like Lyra, that some of his colleagues have been working on that are kind of designed to help people create sort of visual stories more easily. I think there's probably still a lot to do there. Some of the stuff going on at Tableau, I think, is getting into sort of how to help people do things like annotate visualizations, use these sort of basic devices in a way that's just easier than it currently is. So I think that's potentially something we'll see more of. Yeah, there's so much.

I guess that's interesting because I think we've been focusing, of course, we've been focusing so much on the charts, but the context plays such a big role on how you interpret the chart. Right. And studying charts in isolation without putting them back into their own context, I think it's a problem. Right.

Yeah.

And sometimes I even wonder. So one kind of thing that I would love to see is if I give an article to a person, and in the middle of this article, there is one single chart, how many of them look at the chart at all? Right. Or how many times do they go back and forth from the chart back to the text and so on. Right. I have no idea how this works.

Yeah, no, I think that's actually a really interesting area, is how we move between different pieces of information or different types of media when we're looking at a visualization. So, like, I think I could definitely see a lot more being done in that area. Like, how does text go with the visualization? Like, what exactly is the interaction between those things? And is it different for different people.

If I remember correctly, do you mention some of these things in your visualization rhetoric paper where you try to look more at what kind of streaming effects?

Yeah, we actually. We do focus more on the visualization there, though. Like, we do an analysis of a big group of storytelling or narrative visualizations, but we're really focusing on the visualization and less on the text. If there's an article that goes with it, we look at texts such as the titles of the graphs. We found a lot of them were suggestive. You'll get a rhetorical question as the title, which as soon as you see the visualization, it's getting you to think sarcastically about something that's supposed to be true, but isn't actually true based on the data. So that's where we kind of. I mean, that's the sort of text that we look at. But, yeah, I think the stuff I recently looked at with Nick Diapopoulos on the Economist gets a little more at how there's an article and there's a visualization. Sometimes there's also tables and people, when they're trying to sort of reason about the information, are moving from one to the other and focusing on specific parts of, you know, these different representations and trying to bring them all together in an interpretation. So I think, yeah, we don't think enough about that at all in visualization.

Good. I think we should wrap it up. We've been talking for almost 1 hour now. Thanks a lot, Jessica.

Thank you.

Great.

Yeah, it's been fun for me. I've never been on a radio interview, so.

Yeah, it's only Internet. No worries.

Yeah. Okay, well, maybe radio will be next.

Yeah.

Well, thank you very much for inviting me. It was great. Yeah.

Super interesting. I think you bring a really fresh perspective. I really like that. Yeah.

Thank you.

Okay. Thanks a lot, Jessica.

All right, cool. Yeah. Talk to you guys later.

Thanks. Bye.