Episodes
Audio
Chapters (AI generated)
Speakers
Transcript
Visualization Beyond the Desktop w/ Petra Isenberg
Enrico: I'm doing fun projects right now. Like a b infographic analysis software for pro soccer players. And a new type of election map for the German. Data stories number 26 after a long break, summer break.
Enrico BertiniHi everyone. Data stories number 26 after a long break, summer break. Hi, Moritz. Hey, Enrico, how are you?
Moritz StefanerGood, good. How are you?
Enrico BertiniGood. Mellow, relaxed. How do you feel?
Moritz StefanerGood. I had a great summer. In fact was quite sunny here and I took some time off for once.
Enrico BertiniTime off? Time off.
Moritz StefanerActually.
Enrico BertiniTime off sounds weird now.
Moritz StefanerI'm not exactly relaxed, but I feel like in a good mood and productive and so on. So that's cool.
Enrico BertiniCool, cool, cool.
Moritz StefanerYeah, I'm doing fun projects right now, so it's just a small selection of. What I'm doing is a b infographic analysis software for pro soccer players. Like a training device for pro soccer players. And I do the analysis software.
Enrico BertiniWow.
Moritz StefanerAnd a new type of election map for the German. You know, we have the big election.
Enrico BertiniOh, you're gonna have the big elections. When is that?
Moritz StefanerIn two weeks.
Enrico BertiniIn two weeks?
Moritz StefanerYeah. And I'll do like a new type of map for the results.
Enrico BertiniSo you have the chance to influence the elections. Actually.
Moritz StefanerIn fact that's true because we published the results from last time like a week before in this format. And so who knows? Yeah, maybe. Wow.
Enrico BertiniLet's talk about impact. Cool, cool, cool.
Moritz StefanerYeah, well, yeah, it's a nice mixture. What are you up to?
Summer vacation in New York AI generated chapter summary:
What are you up to? Oh, I had a nice summer in New York. My new course started. It's a visual analytics course and it looks like I have some nice students. And lots of new projects going on. Great time, great time.
Moritz StefanerYeah, well, yeah, it's a nice mixture. What are you up to?
Enrico BertiniOh, I had a nice summer in New York. Stayed here, basically walked the kids around. That was fun. I discovered that there are nice beaches around New York, which I didn't expect. Cool. That was fun. That was a lot of fun. And I came back to work kind of like one week ago and was super crazy. But I'm excited. I'm really excited. My new course started. It's a visual analytics course and it looks like I have some nice students. They are excited too. And lots of new projects going on. I have also two new PhD students. So these are actually my first two students here. So I'm super, super excited. So it's great time actually. Great time, great time.
Moritz StefanerYou're getting into the groove.
Enrico BertiniAbsolutely, absolutely. Lots of work. But it's good. It's really good. I like it.
Moritz StefanerSounds great.
Petra IsenbergYeah.
Ventures beyond the Desktop: AI generated chapter summary:
We have another great guest today. We are going to talk about visualization beyond the desktop interface. Petra Eisenberg is a researcher and an old friend of mine. She's been working in the area in the intersection between visualization and human computer interaction.
Enrico BertiniSo let's start. Of course we have another great guest today. We have Petra Eisenberg and we are going to talk about visualization beyond the desktop interface. Petra is a researcher and an old friend of mine and she's been working in the area in the intersection between visualization, human computer interaction, computer supported collaborative work. It's a very nice mix of skills and. Welcome, Petra. How are you?
Petra IsenbergHey, thanks for having me. I'm good. Also had a good summer, really.
Enrico BertiniNot too much taking care of kids too?
Petra IsenbergNo, the grandparents were there for.
Enrico BertiniI don't have that. So Petra, we normally let our guests introduce themselves a little bit. Can you tell us a little bit about yourself? Give a little bio sketch and tell us what you do and what are your research interests or even mention your projects, stuff like that.
Surfaces in the world of visualization AI generated chapter summary:
Petra Kohn: I'm really interested in how we can combine information visualization with novel context or novel devices, and in novel settings, like collaborative settings. There are lots of potential opportunities and of course a lot of challenges when moving from the desktop to surfaces.
Enrico BertiniI don't have that. So Petra, we normally let our guests introduce themselves a little bit. Can you tell us a little bit about yourself? Give a little bio sketch and tell us what you do and what are your research interests or even mention your projects, stuff like that.
Petra IsenbergSure. Yeah. So I started doing information visualization and that I would say is my main area when I was studying in Magdeburg in Germany, a subject called computational visualistics, which is a little bit like computer science with a focus on images. So that was a lot of fun. Afterwards I moved to the University of Calgary to do my PhD with Sheelagh Carpentale. And that's where I actually started to work on this intersection as you just introduced of CCW. So computer supported cooperative work and infovis. And afterwards I did a postdoc at INRIA in France in the Avis team, and that's where I now have a permanent researcher position. So this is my sort of where I came from and where I'm now. And my main research interests are still basically fit exactly the topic of this podcast. So I'm really interested in how we can combine information visualization with novel context or novel devices, and in novel settings, like collaborative settings.
Enrico BertiniGreat. Fantastic. Yeah. By the way, I suggest our listeners to give a look to your webpage because there are lots of interesting examples and images. I guess you also have some videos, right? Yeah, yeah, there are a few, yeah, yeah. We will put the URL on the website, of course. So I think that's really interesting because, I mean, by looking at what you do, it's really interesting because I start asking myself, what can we do with visualization once we move beyond the desktop interface? It looks like there are lots of potential opportunities and of course a lot of challenges. Right. So can you give us a little bit of a broad overview of what happens when you move from the desktop to some other kind of devices? Also, what kind of devices you can move to. I guess you've been working with surfaces, what else? Display walls, stuff like that, right?
Petra IsenbergYeah, exactly. So there actually several, or I would say two main reasons why you may want to move away from the desktop setting. And here I'm saying I'm thinking of the desktop setting as something that the visualization community has considered in most of the past work. And it's sort of a work setting where you imagine there's some sort of analyst who has a lot of data and for him or her, you're creating a visualization to help their work task, whatever it may be. It could be simple charts or it could be something very complex in the area of biology or. So now if you start thinking about surfaces, then basically two new opportunities open up. One is that you can support visualization in new environments. So you can actually think about using visualizations in meeting rooms, in museums, in emergency response settings where there's a really hectic environment, potentially in hallways as ambient displays, or also in shared work settings where several people come together. So that's the one new thing you can consider. And the other one is that you can actually reach new audiences that you may not have been able to reach before. And this is where collaboration becomes really important, because if you think about surfaces that are large. So last time you talked to Dominicus about mobile devices that has some similar new benefits. It's also part of the desktop. But now if you talk about larger surfaces such as tabletops and malls, you can support people standing around the same display and looking at data together and exploring it together and discussing it and making decisions. And of course you can support many non work contexts as well. So audiences in a museum, for example, that may just want to learn something.
Enrico BertiniSure, sure. Well that sounds great. So basically you have a lot of new opportunities because you can let visualization happen in completely new environments, right? And at the same time you have people interacting in completely new ways. Right. That's really, really interesting. Yeah, please go ahead. Sorry.
Petra IsenbergYeah, so this is basically the one type of benefit where you have new users context, either to the location or the types of people you can support. But then of course, if you think about a surface as coupled with the input technology, for example, touched you discussed last time. But there are many, many other ways of interacting with the surface. Surfaces can have more pixels, so you can show more data. You may be able to show data in a completely different way if you think, for example, of a bendable surface or something like e paper and things like that. So there may be very different types of exploration that interactive exploration that you could support with a surface that is not the standard desktop screen with a mouse and a keyboard in front of you.
Tableau and the future of data visualization AI generated chapter summary:
There are visualizations that can happen in this kind of environments and just cannot happen with a standard desktop interface. Which visualization should I never put on a tabletop display like horizontal surface? This is very difficult to answer because it depends on the environment.
Enrico BertiniOkay, so what are. So one thing I'm curious about, I guess there are some type of visualizations that happen in standard environments that make sense to move to this new kind of environments and other type of visualizations that just don't make sense to move there, right? I mean, just as, I guess, and probably the other way around too. I mean, I guess there are visualizations that can happen in this kind of environments and just cannot happen with a standard desktop interface, right?
Petra IsenbergYes. So one of the problems here is that if you're now going to ask me like, which visualization should I never put, for example, on a tabletop display like horizontal surface? This is very difficult to answer because it depends on the environment and depends on the type of task and people you want to support. For example, if you think about something standard like Tableau, for example, and I just take the desktop version of Tableau and put it on a tabletop and put five people around it, most likely it will not be the most effective way for these five people to explore data together, or at least not in a synchronous way. If you have one person driving the interface, then, and the others are just watching, it could still be better than a desktop display, but not for synchronous sort of fluid interaction and exploration in parallel. But in terms of you just think about as a single visualization, then it's a bit difficult to say if you should, for example, never put a lion chart on a tabletop display or something like that because it really depends on who your audience is going to be. If you put something in a museum, you most likely would not put a very complex and complicated visualization for people who will just walk, try and walk up and use it. But if you put it on another type of surface that like in a meeting room where people like say a meeting room where biologists meet all the time to discuss the results from their experiments, then you may be able to put something more complex and very domain specific there that you couldn't put in a museum, for example.
Enrico BertiniSure, sure. So I guess that basically, roughly speaking, we can still have some kind of visualizations that are more some kind of communication kind of visualizations where you expect people to walk by, give a look, maybe have a little bit of interaction and leave, and other type of visualizations that are more targeted towards data analysis. Right. And I think that's a very interesting angle because as soon as you think about what people can do, what kind of data analysis people can do in front of these surfaces or walls, now you have this collaborative kind of capabilities that was very limited on the desktop. Right. So do you have any direct experience of observing people doing some data analysis in a collaborative fashion with this kind of devices?
Petra IsenbergYeah. So we actually did a study, this is while I was working at Microsoft for an internship, we did a study on the Microsoft Surface, which surface? One which was a, I would say like a large desktop display, just as a horizontal surface. So it was a 30 inch and 1024 by 768 kind of display. So not particularly big and not particular high resolution, but you could sit around it with, I would say, to one to four people. So what we did there is we actually gave people a fairly complex task, so they had to search 300 documents for hidden stories. So this was part of the vast contest. I think you interviewed somebody about the vast contest in the past. Within these 300 text documents, they were mostly newspaper articles. There was a story hidden about. So people started with a document talking about a car accident, and somebody was yelling something very strange, like, oh, the flowers. And so people had to go from this accident to actually finding out that there were some illegal things going on in this town, and this guy was poisoned and why he was poisoned and so on.
Enrico BertiniAnd.
Petra IsenbergAnd this task took people around one to 2 hours to complete. So it was fairly complex and not so easy to answer. But what was very interesting was that. So we built a dedicated system for exploring this document collection for people. And we studied fairly regular people who had, like, they weren't researchers or computer scientists, just people basically from any kind of discipline or even seniors or so coming in. And they were very, very comfortable using the system, even though it was for document exploration and search, and fairly complex. Just the familiarity of sitting around a table and being able to look at things together and discussing, it was a great, great entry point. My hunch is that if we had compared it to a desktop setting, that the way people behaved in first entering or looking at this tool would have been quite different.
Enrico BertiniSure, sure. I mean, that's my answer too. And I think so. This reminded me, there is a. I don't know if you remember this kind of study a few years back, I think, 2000 something. There was a study, I think, conducted on standard interfaces from Alfred cops. I guess, that basically says that two pairs of eyes actually discover more stuff than one pair of eyes. So if you put more people on a data analysis task, the sum of what they find is definitely more than what a single person would find. Roughly speaking, I think that was the outcome of the study. And now I think as you move from a desktop interface to something that is more where people can, can actually collaboratively work together, this may be even more powerful. Right. Because you have not only, you have more eyes, you also have the interaction between these people, right?
Petra IsenbergYeah, exactly. So we saw people discussing the hypothesis, challenging each other's hypothesis coming up, and just finding different things. Like you said, they saw more, they could split the work in whichever way they wanted to. And this was really like a benefit to the group work. And interestingly, actually, we also did an analysis of how successful people were in completing the task and how correct the answers were. And those people who worked very, very closely together. So who paid attention to what the other person had read and discussed closely what they had found, they were the most successful. Because even if you use a tabletop display and you are in the same place, doesn't necessarily mean that you always know what the other person is doing. Especially when you're doing very, like, reading tasks where you concentrated on something small, you can't just at a glance see what the other person is actually looking.
Enrico BertiniSure.
Moritz StefanerSo much of the effect comes probably just from the fact that people actually have a dialogue about what they are seeing. And this helps you, like, maybe think in a more structured way or just discover new stuff.
Petra IsenbergYeah, exactly. There are some interface elements that you can build in to help people become aware of what the other person has done. But we need a bit more research to find out exactly if, like, how to design that, such that groups, for example, don't communicate naturally with each other that are actually forced to or strongly encouraged to share more. Like in this document, for example. In this document, example, you could show, okay, the other person has read this, or you have already read this, and the other person has looked at something similar, and you can build interface elements that will show this.
Moritz StefanerYeah, but it's interesting because we are really not used to operating computers together, so. And it's always a strange moment when you sit at your machine and somebody else takes over the mouse. It's always like, what's he doing there? And so, yeah, we're really not used to doing stuff on a computer. Like really? Yeah.
Enrico BertiniIn a shared workspace, in a shared space. Yeah, absolutely. I mean, this level of intimacy that you normally have in front of your computer, this is totally lost, even if you are the single person. So now I'm thinking about the museum scenario. Even if you are the only person who is interacting with the interface, you still have the feeling that it's not very intimate anymore. Right? I don't know, Moritz, I think you've been working on some similar kind of projects where you have some kind of public display somewhere, is that correct?
Public Display at the Venice Biennale AI generated chapter summary:
Moritz Kohn: You have to switch your mindset to a bit more. A polar coordinate system. People can approach the table from any side. You can passively interact by walking closer. Now I'm thinking that there might be also some kind of public displays that are.
Enrico BertiniIn a shared workspace, in a shared space. Yeah, absolutely. I mean, this level of intimacy that you normally have in front of your computer, this is totally lost, even if you are the single person. So now I'm thinking about the museum scenario. Even if you are the only person who is interacting with the interface, you still have the feeling that it's not very intimate anymore. Right? I don't know, Moritz, I think you've been working on some similar kind of projects where you have some kind of public display somewhere, is that correct?
Moritz StefanerYeah, occasionally. I mean, the biggest was probably that table we did for the Venice Biennale in 2008. And it was not a multi touch, but like an object based thing where you could move cards on the table. But that was quite educational, and we sort of started it without much experience in the area and a few things. Well, one thing we learned along the way, for instance, was that if you have a table, you have no up and down, but you have more like center and outside. And this is what you work with. So you have to switch your mindset to a bit more. A polar coordinate system. Yeah, because, I mean, people can approach the table from any side, right. When you have a vertical surface, it's always clear where there's left right, up and down, right. And so that was kind of interesting. And also this whole situation, like, who's in control of the visualization? Or are people competing in some form, stuff like that. That's something you never think about.
Petra IsenbergThere's actually some interesting past research in the era of CCW that that comes from Sheelagh Carpendale. So my previous lab, where they looked at this kind of personal spaces that you form on a tabletop display and what they found, or they call this theory of territoriality, where it's clear that people sort of form these intrinsic spaces around them when they're standing around the tabletop. So you have a personal space that's quite close to where you're standing and then there shared spaces. For example, if the typical example in early tabletop work was this photo collection. So looking at photos together or something, and then if you want to look at something just by yourself, you would move those photos closer to you. If you want to make them available to the whole group, you would move them more in a central location. And then there were some storage spaces that people also often created to make piles of stuff that had a. Yeah.
Moritz StefanerYou want some stuff somewhere, right?
Petra IsenbergYeah, exactly. You need to, like, location is a strong indicator of I need to do something with this.
Moritz StefanerRight. That's interesting.
Enrico BertiniYeah, but so now I'm thinking that there might be also some kind of public displays that are, that I'm thinking about, I don't know, public environments where there is a display somewhere that people cannot really interact with, but past buyers can just look, get some information and leave. Right. I'm thinking about urban environments or stuff like that. That's also an option. Right?
Petra IsenbergYeah. And actually, you can also, you don't need to necessarily think about interaction always as being something active. Yeah, you can. You can passively interact by walking closer. And you could build an inter, like a visualization that reacts just on the proximity of people to the display or something they do, like motion or speech or something like that.
Enrico BertiniYeah.
Big Wall Displays AI generated chapter summary:
Big wall displays could be used in a more data analysis centered kind of way. A perceptual experiment looked at how accurately can people actually read information on a wall, depending on where they're standing. People made quite a few errors, mostly when they had to judge angles.
Moritz StefanerAre there any recommendations, like, for designing these? Because I hear more and more about, like, big display screens somewhere for ambient awareness or let's say big walls at an airport, things like that. Are there any tips you can give, like, from research how. How to work with this totally different situation, like what to avoid or what to encourage? Things like that.
Petra IsenbergSo I've actually worked on big wall displays, but more with the thought of using it in a more really data analysis centered kind of way, because you can imagine having a big wall display that you actually need to analyze data on, whereas what you're describing something in the airport or something you're just passing by the. You would have completely different requirements.
Moritz StefanerExactly.
Petra IsenbergSo I can tell you a little bit about the recent stuff we did on wall displays, but for work purposes, one of the problems we were having is we have this wall here at Inria that is approximately 5 meters by 2 meters. 5 meters long, 2 meters high. So it's much higher than at least I can reach, and it's quite long. So you have to walk, walk along it to see everything. And our question was, okay, if we were to design a visualization for this wall that people actually. Where people actually need to go up and read and analyze the data that we're showing, where should we place this data, and where should we place things like the legends or interaction widgets and things like this? So in order to get some recommendations for this, we went quite low level. So we did a perceptual experiment that just looked at how accurately can people actually read information on a wall, depending on where they're standing. So we said, okay, let's take a very extreme case. You're standing on the very left of this wall, and you're looking at something, and you, for some reason have to read something that's quite far away, like, let's say 5 meters away or above your head. So what we had people do is we showed them, for example, a circle right in front of their eyes and at the far end of the wall, and then we had them match the size of the circle. So this is something that studies how well people are able to estimate the size of objects that are far away and that are at a very oblique angle, viewing angle. And what we found was a little bit.
Enrico BertiniSorry if I interrupt just to make sure. So this basically means that now, since, since people have such a larger extent, I mean, it's much more freedom, you can move close and far away. So now you might actually have new kind of distortions in the visualization. Is that correct? Is that what you're saying?
Petra IsenbergYeah, so, yes, exactly. So, because now you can have all sorts of different viewing angles on the data that on a desktop display, we're currently assuming people looking right at the data, their head is moved just a little bit. But if you're thinking about a wall that's 5 meters long and you're showing a map on it, for example, or, yeah, let's say a map and circles show, for example, the number of people in that city on the map. You could think about just a simple visualization such as this, and you may want to compare the size of this one city to another city. So, in this case, you would have to compare the size of the circles. How well would you be able to do it if the other city is 5 meters away from you? That's the kind of scenario. Does that make sense?
Enrico BertiniIt makes a lot of sense, sure.
Petra IsenbergYeah. So what we found is that people actually made quite a few errors, and mostly when they had to judge angles. So, for example, in the extreme case, when you're looking at an angle, imagine a pie chart. So you have to judge the size of the angle that one pie makes to the angle of another pie that's 5 meters away. People actually thought that, for example, an angle of, say, 100 degrees was 1.5 times the size it actually was. They made quite a bit of an error.
Moritz StefanerAnd they were not aware of that error, or were they aware? Or were they, like, expressing doubt about their own judgment?
Enrico BertiniSo now, well, we asked them to.
Petra IsenbergBe as accurate as possible. So I think they tried. They tried to be quite accurate. But. So the experiment is a bit longer and more complicated. I want to explain in detail. But we had people also. We later gave people the opportunity to walk, to walk closer and double check if they wanted to. And some, we found that things improved when people could walk, but only if they walked in a certain way. If they walked actually to the middle center in front of the screen, they became better. But people are lazy, right? So lots of people just stepped, like, one step back and leaned, you know, as if you're playing basketball and trying to get around someone and then look. But this didn't help at all.
The role of angles in visual design AI generated chapter summary:
Large viewing angles may be a problem. You might want to constrain how people, or giving instructions on how people should move in front of a wall. Also consider if this is potentially a problem for whatever visualization you're building.
Petra IsenbergBe as accurate as possible. So I think they tried. They tried to be quite accurate. But. So the experiment is a bit longer and more complicated. I want to explain in detail. But we had people also. We later gave people the opportunity to walk, to walk closer and double check if they wanted to. And some, we found that things improved when people could walk, but only if they walked in a certain way. If they walked actually to the middle center in front of the screen, they became better. But people are lazy, right? So lots of people just stepped, like, one step back and leaned, you know, as if you're playing basketball and trying to get around someone and then look. But this didn't help at all.
Moritz StefanerOkay.
Petra IsenbergSo, yeah, the large viewing angles may be a problem. And what we also found is you may be familiar with this ranking of visual variables that Cleveland and McGill came up with, where they say, okay, we're really good at judging lengths. So comparing lengths, something we can do almost accurately. And we found something similar for the wall displays. We can judge length quite well. But then, in Cleveland's work, the next best thing would be judging angles and then areas. So the circle example would be the areas. Yeah, we actually found that the area judgments on the wall were better than the angles. So there's something reversed. So I think in terms of guidelines and recommendations exactly for how to design visualizations, there's still lots of research that needs to be done because now we tested something a very low level. But of course, we don't want to just show circles and lines and angles. We want to assemble them into more complex visualizations and then study.
Moritz StefanerYeah, but that's a good piece of information already that you cannot trust your usual rules of thumbnails, necessarily.
Enrico BertiniYeah. And probably also that you need to design also the space in front of your wall. Right. I mean, it's no longer only designing the visualization itself. It's also trying to maybe put some constraints in the space you have in front of the wall. Right. If I understand correctly, I mean, you might want to constrain how people, or giving instructions on how people should move in front of a wall. I don't know.
Petra IsenbergYeah, this is one thing. And for example, if you think about tabletops, of course, you have this weird viewing angle from all sides of the table. You're looking at a skewed angle. So you have to consider if this is potentially a problem for whatever visualization you're building. And if so, then you could move it sort of in a tilted way. I think Moritz worked on a project with the Max Planck Institute where you guys used a tilted table. I'm not sure if this was a design decision you made or who made it, but this is something you can consider.
Moritz StefanerYeah. That was from the company designing the exhibition. And they have developed that as a product. And it's sort of a slightly tilted, maybe 20 degrees or so vertical display. And it's nice because you can stand in front of it and have a comfortable body posture with your hands, you know, and looking a bit down. And you can sort of rest your hands a bit. And you don't have that awkward interaction with, like, a totally vertical surface, which is a bit more demanding, like bodily. That's also something.
Petra IsenbergRight.
Moritz StefanerFatigue. Like you said, people are lazy. And if you run around at a wall and have to move your arms all the time, it is quite. It can make you tired quite quickly. Right.
Petra IsenbergYeah. This is something people are.
Enrico BertiniYeah, this is. This is something I'm really curious about because I've been so. My personal experience with this kind of surfaces is that so when I was in Konstanz, we had this huge power wall. And here, when I moved here at NYU Poly, we also have a wall. It's a little smaller and I am intimidated by this stuff because, I mean, my reaction is the first time you see a visualization on this kind of device, of course your reaction is, wow, it looks beautiful and so many details. But as soon as you try to interact with this stuff, it's a nightmare. I mean, you just don't know how. You don't even know how to start. Right. And I remember, for instance, in Konstanz, we used to interact with, with just standard mouse and keyboard. It's unbelievably hard. Right. And so, Pedro, can you tell us something more about how you interact with this kind of new devices? Because it looks like a big, big challenge to me. And of course, I mean, even if you have new ways of interacting with a wall, like, for instance, gesture interfaces or stuff like that, then you still have new challenges, like fatigue. That is, it's the thing that Morris just mentioned, but probably a lot of other problems, right?
How to interact with a touch screen AI generated chapter summary:
There's not that much research on how to interact most effectively with a visualization using touch. We're actually going to build a very large touch enabled wall here in the future. Can you really isolate the value that the touch brings beyond just measuring speed and accuracy in some task?
Enrico BertiniYeah, this is. This is something I'm really curious about because I've been so. My personal experience with this kind of surfaces is that so when I was in Konstanz, we had this huge power wall. And here, when I moved here at NYU Poly, we also have a wall. It's a little smaller and I am intimidated by this stuff because, I mean, my reaction is the first time you see a visualization on this kind of device, of course your reaction is, wow, it looks beautiful and so many details. But as soon as you try to interact with this stuff, it's a nightmare. I mean, you just don't know how. You don't even know how to start. Right. And I remember, for instance, in Konstanz, we used to interact with, with just standard mouse and keyboard. It's unbelievably hard. Right. And so, Pedro, can you tell us something more about how you interact with this kind of new devices? Because it looks like a big, big challenge to me. And of course, I mean, even if you have new ways of interacting with a wall, like, for instance, gesture interfaces or stuff like that, then you still have new challenges, like fatigue. That is, it's the thing that Morris just mentioned, but probably a lot of other problems, right?
Petra IsenbergYeah. So there's actually a lot of research in particular in HCI that you would, for example, see at the CHI conference, that looks exactly at what would be the best ways and least fatiguing ways to interact with the wall. There are wands, there are gloves. Lots of people now are looking at body tracking, like stuff that they connect, and mid air gestures. And there's some work that takes, as I mentioned before, takes into account where you're standing. So if you're moving closer to the wall, you get more detailed information. If you're moving back, you will be shown something else, or the data moves with you as you move along the wall. It always depends on context, if this kind of interaction would make sense or not. But, yeah, so there's lots of things going on, but specifically for visualization purpose, there's actually very little research on interaction and, well, on non mouse interaction. Like what? Even just something as touch displays. Well, you talked with Dominicus last time, but there's not that much research on how to interact most effectively with a visualization using touch.
Enrico BertiniYeah. And my feeling is that touch is really, I mean, if I have to guess what kind of, if I have to say, what's the kind of interaction I'd like to have? I think this medium sized displays with touch, it looks like the most natural thing I would do. I mean, I never tried, but it looks the most natural thing I would do. I mean, it still allows to have some higher resolution, it still allows you to collaborate with one, maybe another two people, and you have natural interaction in front of it. I don't know. What's your opinion on that? I mean, anything larger than that, it looks really intimidating to me as soon as you try to interact with it.
Petra IsenbergYeah, so we're actually going to build a very large touch enabled wall here in the future and the University of Groningen for example also has one. But I'm not sure quite what the experiences are. And I'm very curious to start some research on, on touch interaction for visualizations on a very large wall. But one thing that needs to be considered when you're just thinking about touch is that it's not usually touch alone that you think about when you're actually implementing a touch based environment. You usually also do away with all the wimp type interfaces, so you're changing. But if you think about something like Excel for example, how you create a chart and how you would interact with a chart, it's still very menus, icons, pointer based, right. On a tabletop or on tablet, usually you would just stop designing and visualization with this kind of interface completely and you're starting to map interactions to other, other things. So you're not, it's not only just adding about adding touch, it's also just making the whole environment different. So it's difficult and there definitely needs to be more work done and that.
Moritz StefanerMakes it hard, of course, from a research point of view to isolate individual factors if everything has changed.
Petra IsenbergYeah, and it's actually also a big problem for validation. So if you're thinking about comparing a mouse based interface to a touch based interface, or let's say you're building a touch enabled scatter plot, something very simple and a mouse based scatter plot, the question is, can you really, in an evaluation, isolate the value that the touch brings beyond just measuring speed and accuracy in some task? What's the value that you get by actually touching data and moving your arms and knowing where in space something is located? It's quite difficult.
Enrico BertiniWow. Yeah, that's interesting. I wanted to ask you something. I forgot, sorry, it was in the back of my mind.
Petra IsenbergI can actually tell you.
Enrico BertiniYeah, go ahead, sorry.
Petra IsenbergI can tell you one more thing that's a little bit related to interaction in that you can also think about designing visualization such that you can sort of passively interact. In one example is a project we're going to present, I'm going to present with Pierre, who you interviewed with Yvonne in the past on the physical visualization. But yeah, so we did a project where we actually combining two encodings of data and overlaying them such that when you close you see one thing and when you far away you see another thing, so you can walk close to the wall, read detailed data, walk far away, and read either completely different or related or just meta information from afar. So this is a way where the encoding is done in a way that without actually having to have a dedicated input device, you can interact by walking close and far, and you still get some different content.
Enrico BertiniOkay.
Moritz StefanerThat sounds like a very, very interesting way of interacting and a very natural one, obviously, because this is what you will do. Like, when you want to learn a detail about something, you will get closer. Sounds nice.
Enrico BertiniYeah. So now I recall what I wanted to ask you. So, moving away from, let's say, the research world, are you aware of any real world settings or companies that are already using this kind of technologies for doing some collaborative work in front of a display? So I remember, for instance, a professor mentioning to me, he's been, he had a contact with a big pharma company, and they are definitely exploring the idea of doing some visualization on large walls to understand, to collaboratively discuss things like molecules for engineering, new drugs or stuff like that. So can you give us, do you have other examples or things you know, or you have experience yourself that are more on the market? Right?
Real world applications of large-scale visualizations AI generated chapter summary:
Moritz: Are you aware of any real world settings or companies that are already using this kind of technologies for doing some collaborative work in front of a display? Moritz: There are two main places. One is museums, one is command and control center. These kind of applications are basically what is called situation awareness kind of stuff.
Enrico BertiniYeah. So now I recall what I wanted to ask you. So, moving away from, let's say, the research world, are you aware of any real world settings or companies that are already using this kind of technologies for doing some collaborative work in front of a display? So I remember, for instance, a professor mentioning to me, he's been, he had a contact with a big pharma company, and they are definitely exploring the idea of doing some visualization on large walls to understand, to collaboratively discuss things like molecules for engineering, new drugs or stuff like that. So can you give us, do you have other examples or things you know, or you have experience yourself that are more on the market? Right?
Petra IsenbergYeah. So I think there are two main places. One is museums, as Moritz probably has some experience, for example, art, and is one of the companies that's making lots and lots of large display kind of visualizations for museums and exhibits and so on. And they're doing really cool stuff. It's worth looking at. And the other one, I would say probably command and control center. So there have even been some research papers describing how people collaborated around a mission, NASA mission planning tool. And recently I attended a workshop on the European Crisis center in Italy. So there were a bunch of companies that came that design, like mission type, I would say, dashboard systems for large displays. So in these cases, they're certainly in use by real world settings.
Enrico BertiniSure, sure. Well, this reminds me, there's also, I'd.
Moritz StefanerLike to mention the panic status board, which is really nice. Panic is quite a nice software company, and they built their own sort of dashboard. They would hang into their own office. It's something, I think it's not done enough. So whenever you have people working together, you could always have, like a big display with information relevant to them. And so they built first the status board and then an iPad app. So you can sort of click together your own little dashboard with information relevant to you.
Petra IsenbergYeah.
Moritz StefanerAnd you can hook it up to a big screen, and then you have your own real time ambient awareness thing. It's kind of nice and I hope we'll see much more of this type of thing.
Enrico BertiniSure. And this actually reminds me that these kind of applications are basically what is called situation awareness kind of stuff. And it's a totally new kind of task for visualization. Right. So we are used to talk about visualizations in terms of either communicating some story or message or doing some deep data analysis. But here you have a completely new kind of task. Right. So you have people in a room who have probably a lot of different displays and they want to make sure that if something happens, they would notice and they would basically figure out as fast as they can what's going on there. Right. And that's a totally new angle and it looks really fascinating to me. Me. So one thing, another thing I wanted to ask you, Petra. So I'm trying to think about our listeners right now. Maybe some of them think, oh, cool, I would do something like that. I would love to do something like that. How can a person start doing that? It looks really intimidating to me, both from the technical point of view, even from the cost. What can a person afford? And knowledge. How can you get the knowledge you need to set up all these things? So it looks a little bit intimidating. So how do I start if I want to do something like that?
How to start a design project with a touchscreen display AI generated chapter summary:
First you need to figure out what scenario and environment you want to support. This influences what types of displays you most likely should be choosing. Also what kind of interaction techniques you may want to supports. How can a person start doing that?
Enrico BertiniSure. And this actually reminds me that these kind of applications are basically what is called situation awareness kind of stuff. And it's a totally new kind of task for visualization. Right. So we are used to talk about visualizations in terms of either communicating some story or message or doing some deep data analysis. But here you have a completely new kind of task. Right. So you have people in a room who have probably a lot of different displays and they want to make sure that if something happens, they would notice and they would basically figure out as fast as they can what's going on there. Right. And that's a totally new angle and it looks really fascinating to me. Me. So one thing, another thing I wanted to ask you, Petra. So I'm trying to think about our listeners right now. Maybe some of them think, oh, cool, I would do something like that. I would love to do something like that. How can a person start doing that? It looks really intimidating to me, both from the technical point of view, even from the cost. What can a person afford? And knowledge. How can you get the knowledge you need to set up all these things? So it looks a little bit intimidating. So how do I start if I want to do something like that?
Petra IsenbergYeah. Yeah. So there are a couple of recommendations I can give, I guess, from my past project. So one of the first things, let's say you're starting from nothing. You don't know what visualization you want, you don't know where you want to put it, or you don't know what who you want to support. So in that case, definitely find the right collaborators first. They can help you with choosing the right technology and helping you with whatever social setting you want to support. But generally I recommend the following process. So first you need to figure out what scenario and environment you want to support. Do you want to put something in a museum? Do you want to put something in a hallway where people will just walk by and give a glance to your visualization and then continue walking? Or do you want to support something where people will actually spend an hour or two in front to analyze data? So this is something you need to know first because it influences what types of displays you most likely should be choosing, what kind of interaction techniques you may want to support, what kind of resolution of technology you may want, and so on. So let's say, should we pick an example? Let's say you want to support something that people will just like an ambient display, people will walk past and they want to see the bus schedule or something very simple. In this case, you need to decide, okay, where do you want to put it? Outside or inside? In a hallway? In a building where it can be destroyed or not. And this influences what technology you want. Should people interact with it or just look at it?
Enrico BertiniSure, sure.
Petra IsenbergSo once you know all of this, then figure out, do you want, for example, a really large display? Are you fine with just a large tv, for example, or a large lcd or a couple small displays? What do you want? Is it really important that you have very high resolution? For example, if you're trying to support someone working in front of a display for hours at a time, I probably go for higher resolution. If it's just for walking by at a glance, you can probably get away with tv resolution. Depends. Something to think about. Yeah. And then you need to consider what type of interaction your surface should support. If you need none, then, well, that's fairly simple. Then you can just pick a tv or an lcd or whatever you want. You don't have to bother. If you want to support touch, then there's a bunch of displays you can buy that already have it included, or you can buy an overlay that you can just hang over your display, like over your tv you bought, or your lcd or something like that. If you want other types of interaction, then you have to worry about, for example, let's say you want like mid air gestures. Then you may want to consider buying a kinect. And where can you put it? Can you put it somewhere so people don't steal it? And I. And things like that. So once you have all of this figured out, like where you put your technology, what it supports, what it looks like and where it will be located, then you should start thinking about your visualizations, what data makes most sense to show to people that, for example, are just passing by, or people like their domain. And for this you may, you may have to follow your typical design cycle of doing some pre studies or whatever needs to be done. Yeah. So then once you have figured out what representation you want, and you want interaction, then you really have to consider how would the interaction make more sense with this representation. So if it's touch based, you have to consider, do you need any menus that people have to click, for example, to set a filter? Or can you just get away with some simple buttons? Can you do any gestures that people would have to learn? Or can you not afford to teach people anything? They just have to be able to use the interface, right? Like as soon as they step in front of it for the first time. So, yeah, it is a little bit daunting, but you learn from every project that you do, and the more experience you get with different types of displays and hardware and software environments and so on, the easier it becomes, in my opinion.
Enrico BertiniAnd I guess there might be some challenges. This is something we discussed with Dominicos as well, some challenges in terms of what kind of libraries you are forced to work with. Right. So how does it look like? I mean. I mean, if you are, I guess you would have to use proprietary libraries or stuff like that.
Petra IsenbergYeah. So it depends a little bit on, similar to what Dominico said, it really depends on if you need a lot of how much data you have and how much power you basically need, you can do a lot of stuff in a web browser. So, like, touch is. Touch, for example, is supported by lots of environments and you can build D3 applications, if you want, that are touch enabled. If you're building a sophisticated, I don't know, brain network visualization, maybe the browser is not the best. And then I. You can, like, there's some projects that, for example, connect things like the VTK library, if you know that one. It's a fairly big one, especially in SCIVIS, scientific visualization with touch. But for example, for the wall display that we're working here, we're mostly working with our own kinds of toolkits. So then it becomes when you have to move stuff to different, like the 16 different machines we have here, then things become a little bit more complicated.
Enrico BertiniOkay. And if I want to get an overview of what are the challenges, the technologies, the existing research, is there any kind of survey or document or anything around that people can use, that you can mention and maybe we can put on the website.
Petra IsenbergYeah, I'm going to send you the link to one with Tobias. My husband originally did a literature review of six or seven different conferences and looked at all the papers that were about surfaces and visualizations in a relatively broad sense. We looked at what data people used in their project, what kind of visual representations they used, what types of displays, what types of input they supported, and sort of what the research focus was and how many. I mean, if they supported single users or groups. So the paper isn't quite out yet, but the Google Doc that contains all the research papers I can give you the link to. Then there's an interest. There are lots of blogs on surface technology. And if you. I can. There's also Google Google list where Johannes Schoening, who's now at Hasselt, called Hasselt University, sends out a newsletter every week with the newest and coolest surface technology. So this is something to subscribe to. I recommend it. I read that every week. And so what else? You asked me three things. I forgot the last one.
What is the best screen for reading text? AI generated chapter summary:
A lot of tabletop displays are not very good for reading text. How many pixels per inch do I need? Do I need to go up to 300, or can I go with 100 or even less? If you have some material there, we will definitely put it into the posts.
Petra IsenbergYeah, I'm going to send you the link to one with Tobias. My husband originally did a literature review of six or seven different conferences and looked at all the papers that were about surfaces and visualizations in a relatively broad sense. We looked at what data people used in their project, what kind of visual representations they used, what types of displays, what types of input they supported, and sort of what the research focus was and how many. I mean, if they supported single users or groups. So the paper isn't quite out yet, but the Google Doc that contains all the research papers I can give you the link to. Then there's an interest. There are lots of blogs on surface technology. And if you. I can. There's also Google Google list where Johannes Schoening, who's now at Hasselt, called Hasselt University, sends out a newsletter every week with the newest and coolest surface technology. So this is something to subscribe to. I recommend it. I read that every week. And so what else? You asked me three things. I forgot the last one.
Enrico BertiniI forgot it, too.
Petra IsenbergResearch. Research and technology. And.
Enrico BertiniYeah, I'm just trying to figure out a person who wants to start with this stuff and what kind of documents are around material that people can use to just get acquainted with the old topic. Just that.
Petra IsenbergYeah. And of course, there's some specialized books. For example, there's one on tabletops, wherever that you can read if you want to learn, for example, about what would be the best resolution I should pick for my display, which is something quite tricky, because a lot of the tabletop displays, and also, if you think about tvs, their pixel per inch ratio is actually not very good for reading text. And reading text is something very important if you have a visualization application. So if you would like to find out, you know, how many pixels per inch do I need? Do I need to go up to 300, or can I go with 100 or even less? Then that would be a good place to look.
Moritz StefanerThat's great, too, because that's. It's true. It's really hard to figure out. And, you know, full hd sounds like a lot, but once you blow it up to a few meters, your text will look really bad, too. So. Yeah, yeah. So that's. That's really a big gotcha in many cases.
Petra IsenbergYeah, yeah. Even the 4k. If you think about a 4k display at 64 inches, it actually only has 72 pixels per inch, which is less.
Moritz StefanerThan your normal screen these days, right?
Petra IsenbergYeah, exactly. You can read ten point text at regular viewing distance, like a desktop viewing distance. Still fine, but anything smaller than ten points already, not that great.
Moritz StefanerYeah, yeah, but that's great. If you have some material there, we will definitely put it into the posts.
Enrico BertiniYeah, yeah, that would be really useful.
Petra IsenbergI'll send you that.
Pedro on Collaboration with Moritz AI generated chapter summary:
Moritz: Pedro is one of my favorite researchers around. At some point, I decided that instead of competing with her, I should collaborate with her. We are doing some stuff together, but it's more on the perceptual side. Is there anything else you want to mention from your work?
Enrico BertiniOkay. I don't know, Moritz. You want to ask something else, or. I think.
Moritz StefanerNo, I think we covered.
Enrico BertiniYeah, I think we covered pretty much everything. Pedro, is there anything else you want to mention from your work? You have so many interesting projects on your website. I got lost basically looking at that. No, seriously. I mean, you guys, you should definitely check her website. There are so many interesting projects and papers all over the place. And I definitely want to mention that Pedro is one of my favorite researchers around. And, I mean, at some point, I decided that instead of competing with her. I should collaborate with her.
Petra IsenbergCome on.
Enrico BertiniAnd it's somewhat true, and yeah, so we are doing some stuff together, but it's more on the perceptual side and not, not related to walls or surfaces. Sorry, we've been running a workshop together. Yeah, that's fun.
What is a good display for a visualization application? AI generated chapter summary:
The time of single touch displays is almost over, but there may still be some around. What would be really nice, and is hardly ever supported currently, is identity tracking. My dream interface would also have haptic feedback.
Petra IsenbergYeah, I guess maybe the only thing that I guess you didn't dare to ask me was about kind of recommending a certain technology. I can, I cannot tell you exactly what I would buy at this moment because it really depends. But what I can tell you is a little bit just the characteristics that I would be looking for that may be potentially important for other people to think about. So from my past project, something that I would really be looking for is that the display is large enough, if you want to support collaboration, that people can stand around such that they don't easily bump into each other, that they can actually have their personal space in front of the display if possible. I would like display to be seamless. So a lot of especially display walls are put together from several, for example, lcd screens. So the least amount of bezels you can have, the better, in my opinion, because especially for visualization applications, it kind of chunks the space artificially or physically. This may sometimes conflict with whatever data you may want to show.
Enrico BertiniYeah, we have this kind of display here and I don't like it at all. When you have these very large bezels, it just doesn't work.
Petra IsenbergAnd as I said before, you want as high resolution as you can get, in particular because for most visualizations you actually need to read labels, so you need to be able to read text and small data items potentially, but you also want high resolution for the input, so you want to be able to interact relatively accurately with your data if possible. I would like multi touch. So I think the time of single touch displays is almost over, but there may still be some around. So the more touches you can have, the better. What would be really nice, and is hardly ever supported currently is identity tracking. So in particular, if you want to be able to show people, for example, what they've already looked at in collaboration and you cannot tell who has interacted where, this may become quite difficult.
Enrico BertiniThat's a very important point that we didn't even mention. Right. I mean, tracking people around, being able. Yeah, from the application standpoint, being able to say who did what, it's, it's a big. Yeah.
Moritz StefanerAnd whose stuff is what.
Petra IsenbergYeah, yeah, exactly. And even sometimes if the fingers, if you think about touch, if the fingers from two people are touching an object of it's the same person can help you resolve conflicts. So yeah, it's true.
Moritz StefanerLike the hands, they might merge if they are too close to each other. Oh my God.
Enrico BertiniWow.
Moritz StefanerHorrible.
Petra IsenbergYeah. So identity tracking something if you can have it. Great. Marker detection is something also very interesting if you want to combine a display of tangibles. For example, if you want to provide some knobs that people can rotate instead of using a virtual interface, or if you want to provide physical data representations on a surface, then being able to track those would be nice. And my dream interface would also have haptic feedback. But this is still a little far in the future.
Enrico BertiniWow. Okay. Directly from the display you mean or what?
Petra IsenbergYeah, there was actually a really nice demo it kind of maybe one or two years ago where it was very small, maybe the size of a phone, but on the display you could touch and by doing some magic you could actually feel the button and it was a flat surface otherwise. So it was really, really nice.
Enrico BertiniWow. Wow.
Moritz StefanerI heard Apple is also rumored to be working on something like touch feedback on there for the iPhone and so on. Yeah.
Enrico BertiniCool. Okay.
Moritz StefanerCrazy stuff.
Enrico BertiniCrazy. It's too futuristic for us.
Petra IsenbergNo, come on.
Enrico BertiniOkay. I think we should stop here. Right?
Moritz StefanerYeah, great, great. Very informative. Lots of great material in the document already, so.
Enrico BertiniYeah, absolutely.
Moritz StefanerIt's good stuff.
Enrico BertiniSo folks, let's just create talking about this.
Petra IsenbergI'll send you a couple more links for the website.
Moritz StefanerFantastic.
Enrico BertiniFantastic. Okay, good. Thanks a lot. Thank you, Petra.
Petra IsenbergGood. Thank you guys. Bye.
Moritz StefanerThanks so much. Bye bye. It.