Data Visualization Literacy with Jeremy Boy, Helen Kennedy and Andy Kirk

I think it is all about diversity, and I think that is the real challenge for designers is that audiences are diverse.

Data stories is brought to you by Qlik, who allows you to explore the hidden relationships within your data that lead to meaningful insights. Let your instincts lead the way to create personalized visualizations and dynamic dashboards with Qlik sense, which you can download for free at click de data stories. That's Qlik Datastories.

Hey, everyone, this is data stories number 69. Hey, Moritz, how's it going? Hey, Enrico, where are you?

I'm actually in a German castle. A schloss. Schloss darkstruck. Yeah, it's a place where mostly researchers, computer scientists can go for a week in a large group. We are like 40 people or something and try to figure something out. And we're figuring something out here.

What are you talking about? What is the main topic?

It's so secret. No, it's about data driven storytelling. So as you know, it's one of my favorite topics. And, yeah, working a bit here on identifying patterns for storytelling and taxonomies and talking about tools and workflows. Very nice. So I hope we can share some of the results on the web. So you're working on a pattern collection. That should be quite interesting. Try to work them into cards that you can use while designing data stories. So it's good. Yeah, and everybody talks about data stories, unfortunately, not the podcast, but storytelling. But we're getting there. Yeah, we're getting there.

We're getting there.

Any pr is good pr, right?

Absolutely.

How about you? How visualization literate do you feel today on a scale from one to ten?

I feel very literate.

So it's an eleven.

Yeah, kind of. So you spoiled it. Okay, well, so today we are going to talk about visualization literacy, which I think we have been mentioning a few times in the podcast already, and so we decided to organize our episode on that. So briefly, very briefly, what is visualization literacy? It's mostly about learning how to read visualization and also how to create visualizations, quote unquote correctly, I would say. So, to talk about this topic, we invited quite a few people this time. I don't know if we ever had three guests at the same time in our show. So we have Andy Kirk from visualizing data. Hi, Andy.

Yeah, kind of. So you spoiled it. Okay, well, so today we are going to talk about visualization literacy, which I think we have been mentioning a few times in the podcast already, and so we decided to organize our episode on that. So briefly, very briefly, what is visualization literacy? It's mostly about learning how to read visualization and also how to create visualizations, quote unquote correctly, I would say. So, to talk about this topic, we invited quite a few people this time. I don't know if we ever had three guests at the same time in our show. So we have Andy Kirk from visualizing data. Hi, Andy.

Good afternoon. Good evening. How are you doing?

Doing great.

Good stuff.

Then we have Helen Kennedy, who is a professor of digital society from University of Sheffield, who is working together with Andy on a research project on visualization literacy. Hi, Ellen.

Hello.

And then we have Jeremy boy, and I'm very happy to have Jeremy on the show because Jeremy is working with me at NYU School of Engineering. He's a postdoctoral scientist and he has done a lot of interesting work in visualization as a communication tool and some research work on visualization literacy. Hi, Jeremy.

Hi, everyone, and thanks for the great introduction.

Okay, so I will ask each of you to maybe briefly introduce yourself and tell a few more words about who you are, what you do, maybe what is your background, and then we can move on to specific projects that you have on visualization literacy. Andy, you want to start?

Sure. Yeah. So, Andy Kirk, I'm a data visualization freelancer based in Leeds, New Yorkshire, and my role today is as a kind of contracted academic for this seeking data project that we've been working on. So obviously I get involved in all sorts of different aspects around consultancy and teaching. And this was an opportunity that arose about two and a half years ago with Helen to have a chance to do some really fascinating research work. So, yeah, it's been a real pleasure to do some proper academic work this last 18 months.

Two years, Helen.

Okay, so yeah, as you said, I'm professor of digital society at the University of Sheffield. I've been researching the digital for nearly 20 years and I'm interested in ordinary people's engagements and how ordinary non experts can be included in different ways. And that's taken me across a range of domains and has often involved working with digital media practitioners, thinking about how the digital products that get made can be inclusive. So I've done a lot of research around web design and web accessibility in the past and I find myself now working looking in the field of data mining and data visualization. As there is more and more data around us, people need to find ways to live with data and visualization is the main way that a lot of non experts access or come across data. I know Jeremy's interested in that term, non experts. I've already used it a couple of times, but that's where my interest comes in, how to kind of open up the world of data to ordinary people.

Great, Jeremy.

Yeah, so I am originally a graphic designer, so my formal trainings in graphic design and I then did my PhD in information and communication sciences where I worked a lot on typically engaging, so I call them casual audiences. But yeah, I'm happy to discuss this term of non experts or everyday people, which I actually quite like what to say. So yeah, now I'm working with Enrico, as you mentioned, at NYU, and we're working with people in the school of Law on Human Rights. Well, data visualization applications for human rights advocacy, which is a really, really interesting topic, obviously with many various challenges, but yeah, okay, great.

So I would like to start directly from the scene data project. This is the project Andy and Helen are working on. Can you tell us a little bit more about what the project is about and maybe also about what kind of results you have produced so far?

Yeah, so for me, the idea for the project started when I was working on another project, which was about how public sector organizations like councils and museums are using or starting to experiment with social media data mining. And we did some of that with that and produce some reports for them of the kind of things that we'd found, which included some visualizations of what we'd found. And I found people saying, oh yeah, I want more of that kind of thing, without even stopping to look at the visualizations. And it made me curious about the really sort of micro level moment when a person looks at a visualization and what happens at that moment. And at the same time I was talking with William Allen from the migration observatory at the University of Oxford about visualisation and the power that it might have. So we went along to a course of Andy's and started talking to him about it and he shared that curiosity around what affects engagement, you know, what makes engagement effective and successful. And again, you know, what happens at that moment when someone looks at a visualization. And this kind of big question of do visualisations work? Are they doing the work that people hope that they are doing? And as luck would have it, at that time a call came out from one of the research councils in the UK for projects about big data and it's Arts and Humanities Research Council. So they really liked the kind of visualization angle. So we got the money and the intention was to find out what factors in the kind of consumption process, the moment of looking at a visualization affect how people engage with it, and also what factors in the production process affect how people engage with it. So some of the research has been looking at, for example, the conventions that are available to visualisation designers and how they enable and also constrain, you know, what can then be visualized. But the bulk of the work has been focused on ten focus groups with just under 50 participants, where we showed them some pre selected visualizations, which we chose to represent a range of chart types, degrees of interactivity, original source or location, subject matter and etcetera, whether it was online or whether it was in print form, and ask people to look at them, asked them to kind of record what they saw, how they felt and what they learnt, and then to talk about those things in a kind of group discussion afterwards. So we had some kind of interviews and other things either side of the focus groups, but the bulk of the research was the focus groups. And from that we have extracted a number of factors that we think affect the process of engagement. So I don't know if you want.

Yeah, so for me, the idea for the project started when I was working on another project, which was about how public sector organizations like councils and museums are using or starting to experiment with social media data mining. And we did some of that with that and produce some reports for them of the kind of things that we'd found, which included some visualizations of what we'd found. And I found people saying, oh yeah, I want more of that kind of thing, without even stopping to look at the visualizations. And it made me curious about the really sort of micro level moment when a person looks at a visualization and what happens at that moment. And at the same time I was talking with William Allen from the migration observatory at the University of Oxford about visualisation and the power that it might have. So we went along to a course of Andy's and started talking to him about it and he shared that curiosity around what affects engagement, you know, what makes engagement effective and successful. And again, you know, what happens at that moment when someone looks at a visualization. And this kind of big question of do visualisations work? Are they doing the work that people hope that they are doing? And as luck would have it, at that time a call came out from one of the research councils in the UK for projects about big data and it's Arts and Humanities Research Council. So they really liked the kind of visualization angle. So we got the money and the intention was to find out what factors in the kind of consumption process, the moment of looking at a visualization affect how people engage with it, and also what factors in the production process affect how people engage with it. So some of the research has been looking at, for example, the conventions that are available to visualisation designers and how they enable and also constrain, you know, what can then be visualized. But the bulk of the work has been focused on ten focus groups with just under 50 participants, where we showed them some pre selected visualizations, which we chose to represent a range of chart types, degrees of interactivity, original source or location, subject matter and etcetera, whether it was online or whether it was in print form, and ask people to look at them, asked them to kind of record what they saw, how they felt and what they learnt, and then to talk about those things in a kind of group discussion afterwards. So we had some kind of interviews and other things either side of the focus groups, but the bulk of the research was the focus groups. And from that we have extracted a number of factors that we think affect the process of engagement. So I don't know if you want.

Me, can I ask one for, did you go into the study with, like, a certain hypothesis or like, a certain idea of what you wanted to have confirmed or rejected? Or was it more like you exposed people to graphics and then recorded, let's say, more exploratively how they would react to them?

Yeah, I think it was. I mean, I'm answering on behalf of Helen here, but I certainly went into it with a very open mind about what we could experience, because I think from my perspective, going back to this kind of conflict of terminology as a. I don't want to call myself an expert, but somebody who's not a non expert, shall we say, I have a certain kind of a tarnished view because I am so immersed in these things already that I don't have that kind of. That naive perspective that many non experts, ordinary, everyday people, do.

So the common people.

The common people, yeah, absolutely. And so from my perspective, there were things I would expect to have seen as factors. There were things I would hope to have seen confirmed. But it was a purely qualitative study that was very much about giving people a chance to respond in whatever way they felt emotionally, both emotionally, but also in much more kind of practical sense, of what did they find interesting? What did they find in terms of insight? So, yeah, it was something that we were very open minded about, really. And that's what the methods were designed to deliver.

So what were the main findings then? Like, what were the main things you found out?

I mean, I think we found out lots of things. But if to answer this question of what affects engagement, what influences how people look at a visualization, that there's a handful of core things, and none of this is kind of mind blowing, none of it is surprising, but I think we do think it's important for maybe, like, the expectations of a visualization designer. Maybe Andy will say a little bit more about that if I just talk about the core things. So the subject matter matters. So, you know, you might, as designers talk about techniques, but if someone's not interested in the subject matter, they're unlikely to stay with a visualization and explore it. The source or location matters. So people, if it's in someone's trusted media, the newspaper they normally buy, or the website they normally look at, they're more likely to trust it, and if not, they're less likely to trust it. We experienced a kind of overall distrust of the media, trying to pull the wall over our eyes or sort of distort things, but that actually broke down a bit. And there was this kind of difference between familiar and unfamiliar media. People's beliefs and opinions mattered, but not only in terms of liking something that confirms your beliefs and opinions. Sometimes it's people liked the experience of having beliefs challenged. So, for example, our case study was migration data and we worked with a migration observatory on that. And some people were surprised at the high number of irish immigrants in the UK because they don't hit the headlines and they enjoyed the experience of having their kind of beliefs around that challenged. A really important thing was whether people feel like they've got the time to look at a visualization. And I think this is because visualizations are still new, they appear in quite new forms, people aren't familiar with them, so they see it as work to have to make sense of and engage with the visualization. So then there was, you know, people needed to feel confident in a range of skill areas. And this is, I think, where we come to sort of define what constitutes visualisation literacy. So they needed to have the language skills. We had a couple of east European community groups who sort of brought up the issue of language skills. Even though we're talking about a visualization, they're obviously often framed in words. People needed to feel confident about their maths or statistical skills, they needed to feel confident about their visual literacy. And some people did and some people didn't. Some people felt confused visually and some people felt confused numerically. They had to have some basic computer skills. And something that came up with people that we interviewed after was this sort of need for critical thinking skills. Both designers, but also participants, said, we need to be able to see what perspective is being prioritized, what's been left out. So there was a kind of consensus there amongst some people that you needed to not just believe what was in front of you, but to kind of think beyond the visualization in front of you. So a really big. I've left the big one to last. And we found that emotions played a really big part in people's responses to visualizations. And they had emotional responses to all of the things that I've talked about already to the subject matter, the source or location, the visual elements, but the data itself, you know, I can see from this visualization that knife crime in my area is going up and now I feel scared. So, yeah, I mean, this is something, you know, that we sort of write about in some of our blog posts that what does it mean for teaching data literacy, statistical literacy and visualization literacy, that emotions play such a big role in people's engagements with visualizations?

I mean, I think we found out lots of things. But if to answer this question of what affects engagement, what influences how people look at a visualization, that there's a handful of core things, and none of this is kind of mind blowing, none of it is surprising, but I think we do think it's important for maybe, like, the expectations of a visualization designer. Maybe Andy will say a little bit more about that if I just talk about the core things. So the subject matter matters. So, you know, you might, as designers talk about techniques, but if someone's not interested in the subject matter, they're unlikely to stay with a visualization and explore it. The source or location matters. So people, if it's in someone's trusted media, the newspaper they normally buy, or the website they normally look at, they're more likely to trust it, and if not, they're less likely to trust it. We experienced a kind of overall distrust of the media, trying to pull the wall over our eyes or sort of distort things, but that actually broke down a bit. And there was this kind of difference between familiar and unfamiliar media. People's beliefs and opinions mattered, but not only in terms of liking something that confirms your beliefs and opinions. Sometimes it's people liked the experience of having beliefs challenged. So, for example, our case study was migration data and we worked with a migration observatory on that. And some people were surprised at the high number of irish immigrants in the UK because they don't hit the headlines and they enjoyed the experience of having their kind of beliefs around that challenged. A really important thing was whether people feel like they've got the time to look at a visualization. And I think this is because visualizations are still new, they appear in quite new forms, people aren't familiar with them, so they see it as work to have to make sense of and engage with the visualization. So then there was, you know, people needed to feel confident in a range of skill areas. And this is, I think, where we come to sort of define what constitutes visualisation literacy. So they needed to have the language skills. We had a couple of east European community groups who sort of brought up the issue of language skills. Even though we're talking about a visualization, they're obviously often framed in words. People needed to feel confident about their maths or statistical skills, they needed to feel confident about their visual literacy. And some people did and some people didn't. Some people felt confused visually and some people felt confused numerically. They had to have some basic computer skills. And something that came up with people that we interviewed after was this sort of need for critical thinking skills. Both designers, but also participants, said, we need to be able to see what perspective is being prioritized, what's been left out. So there was a kind of consensus there amongst some people that you needed to not just believe what was in front of you, but to kind of think beyond the visualization in front of you. So a really big. I've left the big one to last. And we found that emotions played a really big part in people's responses to visualizations. And they had emotional responses to all of the things that I've talked about already to the subject matter, the source or location, the visual elements, but the data itself, you know, I can see from this visualization that knife crime in my area is going up and now I feel scared. So, yeah, I mean, this is something, you know, that we sort of write about in some of our blog posts that what does it mean for teaching data literacy, statistical literacy and visualization literacy, that emotions play such a big role in people's engagements with visualizations?

I think that's very exciting because this matches what we discussed in data storytelling. All these things you mentioned are really important there. And also it's something that is very hard to talk about scientifically. But as you say, it's such a huge part of how people actually engage with information or experience information in real life settings. Right? And a lot of the data visualization research, of course, has focused on exploration and analysis done by experts. And here we have the total opposite. I think it's very, very exciting and great that you tackle that.

I mean, one of the most kind of influential methods that we used in the focus groups was this technique called talking mats, where we use this, essentially this two by two grid to capture people's general assessments. And obviously that's what it could be about, the degree to which they liked or disliked the thing, the experience, the entire kind of engagement. And to what degree did they feel like they'd learnt something or not learned something? Now, obviously, learn is not necessarily the best word because not every visualization has to teach you something new. It can confirm or reinforce what you already knew. But it was the language that people understood. It's the language that people kind of know how to make some very quick assessment about how they're feeling. And once again, in one of the blog posts that we wrote about the project, we shared some of the kind of scores that people gave for some of the projects that we use. And you can see the diversity of opinions, whilst there are some that do have a sense of a general consensus, as it were, you can see the diversity of likes and dislikes and learnt and didn't learn. I mean, obviously, in an ideal world, we would have had thousands of participants and we'd be able to dive into the demographics and the characteristics of the people to try and tease out the reasons why. But as you say, Moritz, it's one of those things that it's not a quantifiable thing, it's not a scientific thing that a lot of people in the field are more comfortable with in terms of discussing and acknowledging. You know, things like appeal is such an ambiguous, elusive term, but it's the reality. And that's kind of what we've hopefully arrived at, which is confirmation of these important human factors.

So one aspect that I'm really interested in is, I think there are two things. One is whether people are able to recognize a given chart, right, even before being able to extract knowledge out of it, just being able to recognize, to know how to read the chart in the first place. I think that's an interesting aspect. And the second one, once you know how to read something, whether you are able to extract information correctly out of it. Right. So I'm wondering, did you notice anything in your studies that is related to these two aspects? Like, is there anything that people, any kind of charts that people just didn't know how to interpret? And then I would be curious to hear what is the reaction there. Right. And second, whether you've been checking on whether people just don't extract the right information out of them, because to me, these look like two very important aspects of visualization literacy.

I mean, there was certainly. So, just to give you a couple of examples of the projects that we expose people to. First of all, we have the classic stream graph, the ebb and flow of box office receipts from the New York Times. A very unusual graph to use, which is hard to use, right. But it is accompanied with some good annotation assistance. It explains the colors, the bandings, the sizes. So it kind of gives you the assistance to kind of penetrate past that unfamiliarity with that chart type. There were some people who commented on still not being able to understand the significance of the colors, the significance of the up and downness of the chart, even though they are given the guidance. And I think there's part of that, which is the patience that we, as people tend to not have for having to learn something new, especially if we don't necessarily see the immediate payback of putting in those efforts to learn how to read it. We had a kind of a Sankey diagram, a two sided sankey. This was a project from Scientific American looking at the consumption of water around the world and the usage of water. And once again, people were quite confused how to join the lines together because it wasn't interactive. So you had all these overlapping lines with a relatively ambiguous z sorting. And people did struggle to make their way through that tangled web of lines. We add monitors and cos better life index the flowers totally which. Reasonable, but really nice, absolutely impenetrable. But it's an unfamiliar prospect. But the familiarity of the metaphor of the flower once again gets you over that. So I guess to answer your question, in amongst this deluge of qualitative material we've got, there were definitely comments about people succeeding with reading new charts, struggling to read new charts, struggling to read charts that we would have expected them to have no trouble with. So once again there didn't seem to be any commonality in people. Because we don't really also know their history before this way of exposure to different examples. But yeah, it's something that once again we would look to once again revisit all the qualitative material to find out more information. But it certainly plays a part. But there were so many other things that people. People talked about. Not necessarily always explicitly. I didn't know how to read that chart.

I mean, there was certainly. So, just to give you a couple of examples of the projects that we expose people to. First of all, we have the classic stream graph, the ebb and flow of box office receipts from the New York Times. A very unusual graph to use, which is hard to use, right. But it is accompanied with some good annotation assistance. It explains the colors, the bandings, the sizes. So it kind of gives you the assistance to kind of penetrate past that unfamiliarity with that chart type. There were some people who commented on still not being able to understand the significance of the colors, the significance of the up and downness of the chart, even though they are given the guidance. And I think there's part of that, which is the patience that we, as people tend to not have for having to learn something new, especially if we don't necessarily see the immediate payback of putting in those efforts to learn how to read it. We had a kind of a Sankey diagram, a two sided sankey. This was a project from Scientific American looking at the consumption of water around the world and the usage of water. And once again, people were quite confused how to join the lines together because it wasn't interactive. So you had all these overlapping lines with a relatively ambiguous z sorting. And people did struggle to make their way through that tangled web of lines. We add monitors and cos better life index the flowers totally which. Reasonable, but really nice, absolutely impenetrable. But it's an unfamiliar prospect. But the familiarity of the metaphor of the flower once again gets you over that. So I guess to answer your question, in amongst this deluge of qualitative material we've got, there were definitely comments about people succeeding with reading new charts, struggling to read new charts, struggling to read charts that we would have expected them to have no trouble with. So once again there didn't seem to be any commonality in people. Because we don't really also know their history before this way of exposure to different examples. But yeah, it's something that once again we would look to once again revisit all the qualitative material to find out more information. But it certainly plays a part. But there were so many other things that people. People talked about. Not necessarily always explicitly. I didn't know how to read that chart.

I mean, I think one of the things that Andy's talking about is time. I think lots of chart types are unfamiliar to people. But when they were given the time in our focus groups, they lasted 2 hours. The focus groups. The first half was looking at charts and the second half was talking about the experience that on the whole people did manage to make sense of it and did manage to extract information correctly. Except there was one incidence in one focus group where people were quite young, they were young farmers. That was their kind of collective group where they misunderstood some information in one of the visualisations that we showed them. I think people are, apart from the obvious bar chart and pie chart, generally unfamiliar with a lot of the visualization types that are out there. But given time, can generally make sense of them. Now, they might be put off by them. I think a lot of people were put off by the aesthetics of the ebb and flow. And so they didn't want to invest the time. So that's a kind of emotional, kind of gut reaction. Some people who were more visually oriented were more compelled by the graphics or the visuals of ebb and flow. So in answer to Enrico's questions, I think people can read charts and can extract information correctly given the time. And Andy was talking about the patient, people's patients. But it is a question of. We're talking generally about visualization shared in the media because that's generally where people who aren't experts dealing with data in their jobs come across them. And there is an expectation that media can be consumed quickly. And that's not the case with some kind of complex or deep visualization.

Just a couple last points. Just two last points. But one thing to make clear is that we didn't assess people's comprehension of what they were reading. We didn't say what is the answer to this? Can you find out which is bigger? It wasn't that level of quite mechanical observations. And secondly, and I think it's something that you've spoke about before, Enrico, about the idea that some people enjoy the task of a puzzle. They enjoy the task of trying to accomplish something that looks difficult and unusual. And obviously in the artificiality of a focus group, they were given that chance to work through this unusual form to get to the other side, hopefully. So that was something that people at times did mention in the comments.

So this is a great time to take a little break and talk about our sponsor this week. Once again, data stories is brought to you by Qlik, who allow you to explore the hidden relationships within your data that lead to meaningful insights. Let your instincts lead the way to create personalized visualizations and dynamic dashboards with Qlik sense, which you can download for free at Qlik deries and this week, the Qlik blog features a new blog post on data visualization maps and the literacy required to read and interpret them. It's called Hibi Dragons and presents a few really interesting non standard maps. For instance, it features the wonderful Galpidus projection. It's an equal area projection that seems upside down. So Africa is on top and Europe on bottom, and you're kind of confused by it. But of course the earth is a sphere and in space there is no right way up or no natural up and no natural down. But we are so used to our standard views of the world, this map suddenly is very intriguing and almost makes you a bit nervous when looking at it. The blog post also features Buckman's the fullest brilliant dymaxion maps. We see hand drawn maps and in the end, even a physical stick map of tied together sticks representing shipping routes in the Marshall Islands. And it closes with these two really good questions. And the one thing is when you create a map, always ask yourself, will my audience know how to read this map? And second, what else is this map communicating? So thanks again to click for sponsoring us this week. Check out the blog post. The link is in the show notes. And now back to the show. Can I move the discussion to one thing that I think is a bit of the elephant in the room, to me at least, is the topic of diversity. Because I think there's like the first thing when we look at these studies about literacy is the first thing you realize is, wow, there's huge intelligence, interpersonal differences in how people perceive charts, how they react to them, how much they engage. And as you said, the results can be all over the place. Like somebody might love one chart and hate the other one, and for the other person, it's the other way around. How much do you think does that have to do maybe with that? The people producing charts are from a specific demographic and are specific types of people, like white male dudes in their thirties, like me. And maybe there's like a total lack of sensitivity to the actual diversity of the audience. Just throwing that out there. What's your take on that?

So this is a great time to take a little break and talk about our sponsor this week. Once again, data stories is brought to you by Qlik, who allow you to explore the hidden relationships within your data that lead to meaningful insights. Let your instincts lead the way to create personalized visualizations and dynamic dashboards with Qlik sense, which you can download for free at Qlik deries and this week, the Qlik blog features a new blog post on data visualization maps and the literacy required to read and interpret them. It's called Hibi Dragons and presents a few really interesting non standard maps. For instance, it features the wonderful Galpidus projection. It's an equal area projection that seems upside down. So Africa is on top and Europe on bottom, and you're kind of confused by it. But of course the earth is a sphere and in space there is no right way up or no natural up and no natural down. But we are so used to our standard views of the world, this map suddenly is very intriguing and almost makes you a bit nervous when looking at it. The blog post also features Buckman's the fullest brilliant dymaxion maps. We see hand drawn maps and in the end, even a physical stick map of tied together sticks representing shipping routes in the Marshall Islands. And it closes with these two really good questions. And the one thing is when you create a map, always ask yourself, will my audience know how to read this map? And second, what else is this map communicating? So thanks again to click for sponsoring us this week. Check out the blog post. The link is in the show notes. And now back to the show. Can I move the discussion to one thing that I think is a bit of the elephant in the room, to me at least, is the topic of diversity. Because I think there's like the first thing when we look at these studies about literacy is the first thing you realize is, wow, there's huge intelligence, interpersonal differences in how people perceive charts, how they react to them, how much they engage. And as you said, the results can be all over the place. Like somebody might love one chart and hate the other one, and for the other person, it's the other way around. How much do you think does that have to do maybe with that? The people producing charts are from a specific demographic and are specific types of people, like white male dudes in their thirties, like me. And maybe there's like a total lack of sensitivity to the actual diversity of the audience. Just throwing that out there. What's your take on that?

Well, I'll speak as a non white male dude in my thirties. You're producing different kinds of visualizations despite your similarities. But I think it is all about diversity. And I think that is the real challenge for designers, is that audiences are diverse. And I think Andy came away from the project sort of at ease with the idea that you're never going to appeal to everybody. You know, there's maybe a 50% max, you know, that you can inspire by visualizations. But in a way, I think some visualization research, you know, hasn't kind of acknowledged that point that the audiences are diverse. And so what do we do with that? And how can we, how can we kind of address that in the visualizations that we make?

Yeah, and my feeling is that what is perceived as being intuitive or easy to read, you know, if you just ask your five friends who are like doing exactly the same thing as you, you might get it wrong. I'm just saying.

Yeah, just the last point to make. I mean, obviously we had a smallish group, but we had a real diverse set of participants, non English speaking in their first language, north south artists, data people, farmers. So we deliberately tried to handpick very different groups to try and tease out those differences. Obviously not statistically large enough a sample to draw any conclusions from. But hopefully it helped to bring out the diversity of opinions raised.

So I think maybe that's a good time to move to Jeremy a bit, because Jeremy has been trying to measure literacy formally. Right. So I think, I'm curious to hear from you. Maybe you can briefly introduce your work, what you've done, and also talk about this problem of diversity, because do you see diversity in your data and how do you handle that?

Great question.

Sorry for the hard question.

But not yet. So, to talk about the work, I think the best kind of beginning question or introduction is the question you just asked before on the difference between being able to directly, immediately extract information from a chart and then being able to gain higher insights from the whole visualization or the whole infographic piece. And so what I studied was specifically, can people extract information from common charts like bar charts, pie charts and so on. And are they confident about extracting this information? And so what we did is we ran a series of studies with different types of charts on Mechanical Turk. And what we found is that a lot of turkers actually do have trouble just, you know, looking for simple things, actually, like finding averages and things like that. So what, the interesting thing is that they are able to do it, but they're very, they're not confident. They really lack confidence. And so, yeah. So for this question of diversity, I don't think we found anything specific in our results like clear cuts between generations or between types of populations. But then also maybe that our method wasn't most appropriate for that because we used item response theory and it's hard to correlate back to demographic data.

Yeah. And I think you told me so when we started this project. I think the project stem from the fact that you realize that in previous studies there are many people who are actually not able to answer to any of the questions.

Right.

They just don't know how to read the chart in the first place. Yeah.

So that was the, the initial motivation, basically. We had run a couple of other studies with Jean Danyel Fiquette, my supervisor at the time. And we had found that for even simple things, people were reacting very strangely. And it sort of came to mind that maybe, hey, this is a big problem. They don't even know how to read these charts that we give them. And it turns out that, yeah, it is typically on mechanical Turkish.

Yeah. I think what I haven't seen so far, that would be really nice to assess. It's kind of like trying to understand what are those charts that most people do comprehend. Right. Where do we draw the line? Right. Is everyone able to extract information correctly from a bar chart? Well, actually, from your study, it looks like the answer is no. But I mean, which is, is somewhat troublesome. Right.

How much lower can they go?

Yeah, exactly. And I remember some years back, I remember people from New York Times mentioning that they would never use a scatter plot, at least for many years, because people just don't know how to read a simple scatter plot. Right. I think that's a very interesting problem because we have, on the one hand, probably a very large segment of the population who is not able to extract information from a bar chart correctly. And on the other hand, we have a substantial number of people who are producing a lot of really complex visualizations out there. And of course, the result shouldn't be that we should stop doing complex visualizations but, I mean, I think we can agree that there is some interesting problem there, right. And some challenges.

Some of the problems are about people's feelings about numbers, I think about data and about statistics, which inspire fear and anxiety in a lot of people. And that comes down to particular kind of cognitive, rational ways of teaching about statistics and statistical literacy that, you know, in my privileged position as an academic researcher, I might be able to kind of experiment with more novel and inventive ways of teaching statistical literacy. Then getting that sort of taken out into curricula would be a big challenge. But at least we could learn whether different modes of introducing people to data, numbers and statistics, might inspire more confidence, which then might make people more confident looking at a visualization. I really do think that the context matters as well. There's a colleague here at Sheffield who I know did a study in a sort of business context, in a kind of work dashboard context of which types of charts people could best extract information from. But to make it interesting for participants, he gamified it, and you sort of got brownie points. If you did it quickly, I think that you're going to get more error then, you know, our kind of sort of cozy way of his and our to look at eight visualizations actually inspired quite a lot of apparent understanding, and they were really quite diverse and some unknown chart types there. So I think if you say, here's 50, do it quick and you'll get a star, or you say, here's eight, take your time. There are no wrong answers here. You kind of get different results.

Yeah. I wanted to come back on the question of confidence. So you're talking about confidence with numbers. But I think what we saw, confidence in your visual or perceptual system, people could kind of deal with the numbers, but they wouldn't necessarily trust their visual system for doing so. And so there are definitely these different things between data literacy, I'd say, for the confidence with numbers, with trends, with things like this, the confidence in your visual system, which will be visual literacy literacy, maybe to some extent visualization literacy, and the connection between both, which I think is specifically visualization literacy.

Absolutely. I mean, just going back to the starting point for me in my motivation for getting involved in this project was largely driven by my experiences on my training workshops with everyday people, again, but everyday people with perhaps an interest in visualization, not two everyday people, but. And the fact is, when we were discussing, because a lot of the exercises I do involve reading and critically evaluating charts and a lot of the discussions, I've always had with people is we don't get taught how to do this. We don't get taught how to do this at school, we don't get taught how to do it at university. We get by through exposure, we get by through practice. And what we've tried to do in the project, as one of the small outcomes of it is, or sorry, outputs of it, is an attempt to just give people a very simple reading strategy to help them make sense of a visualization more effectively, but also more efficiently knowing that time and pressure is a key dynamic of a situation. Because I'm sure we're all the same, even though we're very familiar with visualizations. When I first see something pop up, I'm straight into the content, I'm straight into the big and the small. I'm looking at the big, flashy things. I don't just stop, take a breath, read the introduction, read the title, look at the axis scales. And it's just like when you get a new toy, a new tool, like a new phone, you don't read the mind, you just, you dive in, you play and you make mistakes.

Then you complain that it doesn't work.

And then you complain that, yeah, why does this not work? So it's very boring to tell people to read it, to look very carefully through all these stages, but it will give them all the explanatory aids they need to have a better chance. But it's a shift in mindset and it needs to go back to education.

So it's a bit like driving school, but for charts.

Exactly, yeah.

There we go. I have another question. I think this, I'm not good at maths feeling is a really crucial one that for somebody was like, oh, it's a chart, I'm not good at that. In your experience or your studies, do you feel you can say that metaphors help because it's a common intuition among designers that natural metaphors that present a good mental model of how to think about something, or the use of analogies in teaching. Right. Like an atom is like a planetary system. It's not quite, but it helps in getting a few of the ideas across. Do these things help in your view or have you investigated that?

There we go. I have another question. I think this, I'm not good at maths feeling is a really crucial one that for somebody was like, oh, it's a chart, I'm not good at that. In your experience or your studies, do you feel you can say that metaphors help because it's a common intuition among designers that natural metaphors that present a good mental model of how to think about something, or the use of analogies in teaching. Right. Like an atom is like a planetary system. It's not quite, but it helps in getting a few of the ideas across. Do these things help in your view or have you investigated that?

We didn't really look at that, so I couldn't say anything general about our findings. I think, as we've already noted, different visual styles appeal to different people differently. So the flower metaphor on better life index appealed to some people. Quite a lot, actually. I think there's quite a lot of in the like, square there. I don't know about Jeremy, but we.

Also looked at, we didn't do this in the focus groups, but we looked at the idea of semiotic analysis of visualizations and look at the metaphors and the color connotations and things like this. And whether they are intended or not is another matter, but it is something that we as readers or perceivers do process. And I was surprised, for example, that more people didn't quite get to grips with the stream graph. And I said, I suspect it's because they were looking for exact values, they were looking for very precise readings rather than the general sense, as the title supposes, of this seasonality, this ebb and flow. And I think sometimes, I think we maybe as a society, we're so consumed by a desire to look at detail that we struggle to have that relaxation of just saying, okay, a gist is fine. You know, you can't judge the size of the petals on the better life index flower with a degree of accuracy, but you get a center, the big, the small, the medium. You're a click away from the bar chart view, which gives you that extra degree of readability. But I think some people are so driven by need. I want the exact, I want the numbers. I don't want the gist. I want the numbers.

I'm surprised to hear that, Andy. I'm surprised to hear that. I think my sense is that most many people react emotionally to the aesthetics of side of every visualization. Right. And when you have a colorful stream graph with, I don't know, bells and whistles, you have people saying oh, that's a really cool chart, I love it. And then you see. And there are lots of problems there. Right? So I think it's an entry point.

Yeah, absolutely. But then I think you kind of push through that 1st, 30 seconds or so when they're on board and then it's not the so walking.

I think maybe because it's because it's a visualization of data, people might expect precision. But one of the really kind of powerful moments of learning about the role of emotions was after the focus group. We selected some people to keep a diary for a month of their everyday encounters with visualizations. And we interviewed them at the end of that period. So about five or six weeks after the focus group. And we asked them if they could remember any of the numbers from the visualizations in the focus groups, any of the actual data, and they couldn't, but they could remember the feeling. You know, I remember feeling surprised that that number was higher than I expected or that that number was really low. And that seems to be the important message, you know, that it's not remembering 24 71, it's, wow, that was surprisingly high. Or that was kind of troublingly low. So I think that's maybe the bigger picture. Aim here rather than communicating actual numbers, although those feelings are based on actual numbers.

I think that's a very interesting point, Helen, that you raised here, because I see a tension between the way people actually process information out of visualizations and the way we design visualizations. So what I want to say is that if you look at that little bit of theory that we have on how to design visualizations, quote unquote properly, you would see that most of the guidelines are centered around the idea of effective channels, effectiveness. Right. But then what you find when you look at how people process information and what they remember is that they remember the gist of it. Right. They remember the message. They don't remember what you just said. They don't remember the number, they remembered the message. Right. So, yeah, I mean, I think it's just a comment that I find this tension between this or maybe mismatch between these two things. And I find it interesting.

There's a lot still to research on the communicative side of data virtualization. I can attest that also here from.

The virtual experience from the workshop. Yeah, yeah, yeah. There's so much more. Yeah. Ellen? Yes.

Yeah. Can I just follow you? I think some of the themes that have come up in the discussion, like about diversity of audiences, about how you measure whether people can interpret data correctly from visualizations, makes me want to plug the fact that following on from the scene data project, we've managed to get funding for three PhDs in the field of data visualization. One is about visualization literacy, one is about sort of responding to diversity when making visualizations, and one is about sort of measuring effectiveness. Trying to take that talking map that Andy talked about and turn it into a widget that everyone can use to get some immediate feedback. So I think Andy's going to put a link on his visualizing data site. Unfortunately, there are some conditions around who can access the full funding and who might be able to access of fees, only funding. So sadly, it's not available to everybody. But, you know, I think we're all really interested in more research in all of these areas, and this is opportunities for any listeners out there that might want to take up that.

Yeah. Can I just follow you? I think some of the themes that have come up in the discussion, like about diversity of audiences, about how you measure whether people can interpret data correctly from visualizations, makes me want to plug the fact that following on from the scene data project, we've managed to get funding for three PhDs in the field of data visualization. One is about visualization literacy, one is about sort of responding to diversity when making visualizations, and one is about sort of measuring effectiveness. Trying to take that talking map that Andy talked about and turn it into a widget that everyone can use to get some immediate feedback. So I think Andy's going to put a link on his visualizing data site. Unfortunately, there are some conditions around who can access the full funding and who might be able to access of fees, only funding. So sadly, it's not available to everybody. But, you know, I think we're all really interested in more research in all of these areas, and this is opportunities for any listeners out there that might want to take up that.

That's fantastic.

Yeah. I think my wrap up point was just a sense that this project is certainly not the end. It's whetted the appetite. It's opened up a few ideas about what to look at next. Give us a few. Give us a bit of a structure of the things look at. But yeah, there's so, so much more to look into.

Yeah. And I have to say I'm glad to see that a few research papers on this topic are popping up. I think last year there was an interesting one at infovis on how people interpret visualizations they're not familiar with. So that's an interesting one. And so things are developing and I'm really curious to see what is going to happen and what new knowledge we are going to generate in the next few years. I think that's a very, very important topic. I just want to raise one last issue that I want to mention, and it's about, I want to go back to the idea of being able to read charts correctly because I believe that's a super important point when we consider the problem of deception. So it's so easy for, and I mean, Ellen, you said something like people just very quickly look at a chart and extract a message out of it. Right. And this is what happens most of the time in the real world. Right. So the problem is it's so easy to create even the most basic chart and in a way that people get the wrong message but still get this sense that whatever is presented there must be somewhat true because it's based on data. Right. So again, we have another interesting dichotomy, right. Between the fact that people perceive anything that is based on data as the truth. And on the other side, it's so easy to mislead people with even just the most basic chart. Right. I don't know if you remember the planned parenthood one. It was just two lines. Right. It's like, it's crazy.

Yeah. And I have to say I'm glad to see that a few research papers on this topic are popping up. I think last year there was an interesting one at infovis on how people interpret visualizations they're not familiar with. So that's an interesting one. And so things are developing and I'm really curious to see what is going to happen and what new knowledge we are going to generate in the next few years. I think that's a very, very important topic. I just want to raise one last issue that I want to mention, and it's about, I want to go back to the idea of being able to read charts correctly because I believe that's a super important point when we consider the problem of deception. So it's so easy for, and I mean, Ellen, you said something like people just very quickly look at a chart and extract a message out of it. Right. And this is what happens most of the time in the real world. Right. So the problem is it's so easy to create even the most basic chart and in a way that people get the wrong message but still get this sense that whatever is presented there must be somewhat true because it's based on data. Right. So again, we have another interesting dichotomy, right. Between the fact that people perceive anything that is based on data as the truth. And on the other side, it's so easy to mislead people with even just the most basic chart. Right. I don't know if you remember the planned parenthood one. It was just two lines. Right. It's like, it's crazy.

At the same time, people think it must be not true because it's in the media time. It must be true because it's based on data and it must not be true because it's in the media and people manage to have those two contradictory thoughts at the same time.

That's a very interesting angle.

Maybe the next study should look at cognitive dissonance.

Yeah, exactly.

Excellent. That was really fascinating. I'm really glad this type of research is happening. I can really say that. And I'm looking forward to read the report. We'll link to the, there's a good summary of the results on visualizing data.com, comma. We'll link to that and any other resources and. Yeah. Thanks so much. Fascinating. Thank you. So we should have you back in two years and see what the next stage of the research has brought.

Yeah, sure.

Absolutely.

Also from Jeremy, thanks so much for coming. And talk soon.

Bye bye.

Yeah, bye. Thank you.

Bye bye.

Thanks.

Bye.

Hey, guys, thanks for listening to data stories again. Before you leave, we have a request if you can spend a couple of minutes rating us on iTunes, that would be extremely helpful for the show. I also want to give you some information on the many ways you can get news directly from us. We are, of course, on twitter@twitter.com. Datastories. We have a Facebook page@Facebook.com, data storiespodcast. And we now also have a newsletter. So if you want to get news directly into your inbox, go to our homepage, datastory es and look for the link that you find on the right. One last thing I want to tell you is that we love to get in touch with our listeners, especially if you want to suggest way to improve the show. Amazing people you want us to invite or projects you want us to talk about. So do get in touch with us. That's all for now. See you next time. Thanks for listening to data stories. Data stories is brought to you by Qlik, who allows you to explore the hidden relationships within your data that lead to meaningful insights. Let your instincts lead the way to create personalized visualizations and dynamic dashboards with Qlik sense, which you can download for free at Qlik Datastories. That's Qlik Datastories.