emoto (with Stephan Thiel from Studio NAND)

Hi, everyone. Data stories number eleven. Hi, Moritz, how are you?

Hi, Enrico. I'm doing great, thanks. How are you?

Good, good, good.

You were not prepared for that question.

No, no. There is still some sun outside and. Yeah, it's fine. It's fine.

Cool. You're still in Europe, which is.

I'm still in Europe, which is good.

For Europe, but bad for you.

Yeah. I don't know if these Americans really want me. Yeah. There is a delay in the visa process and I'm living in an empty apartment with my wife and three kids, one of whom is an infant. And it's a lot of fun, as you can imagine.

Finally enough room to play.

Yeah, yeah. I took my inflatable bed from the basement and it's. Yeah, the kids really love it. How are you? How's it going?

I just came back from England from the cave. We set up the final sort of chapter of the Emoto project, which we will talk about today.

Yeah.

And it was about visualizing the response to the games, London 2012, and just came back from setting up the installation part, sort of. We had an exhibit, physical exhibit and. Yeah, now I'm catching up. And next week. Oh, no, end of this week, going to Helsinki for the data cuisine workshop and open knowledge conference. So that should be fun too.

So did you have any holidays or anything that resembled.

No, not really. I mean, this year was a bit unlucky with the holidays because of the Olympics, which were like right in summer, but I'll try and take a week off, end of October or something like that. I mean, I have lots of conferences now which are in a way also holidays.

I am myself in this strange situation. I didn't have any real holidays because we were basically spending our holidays packing and selling stuff. We sold almost everything from our apartment. And then we discovered. Yeah, we had holidays on eBay. Yeah. Fortunately, Konstanz is quite nice. We have a lake that is quite close from our apartment, so we had some fun around.

That's something.

Yeah. And what else? Yeah, I'm working on some this week stuff. This week is approaching quite soon and. Yeah, it's quite nice. We have. I'm organizing a workshop together with some people on visualization evaluation and it's quite. I think it's going to be quite cool. Fourth time that we organized that. And the workshop is called believe and I don't even remember anymore. What's the acronym for? It's more a legacy of the first one. It's quite convoluted kind of acronym. And. Yeah, I think it's going to be cool. And I'm also organizing a panel there on reproducibility in visualization research, which I think it's interesting for people, not only for academics, but also for people like you, Moritz, who, as I know, are interested in accessing not only papers, but also code and nice stuff from research. So I think it's an interesting topic to discuss there.

So the idea is that people publish not just the charts, but also the whole, everything. You need all the ingredients to reproduce. Yeah, visualization, I think that's a great idea.

And there is actually a quite strong trend in computer science in general. Like, for example, in databases. At the SIGMOD conference, which is basically the leading conference in databases, they have this reproducibility initiative since two or three years, and they are quite advanced right now. So there are some niche niches where reproducibility already, it's already at a good stage. And some others, like visualization, where we are basically not too advanced. We are not even discussing too much about that yet. Anyway, the episode is not about that. We are going to talk about Emoto.

That's true.

Which is. Yeah, maybe. Moritz, you want to introduce Emoto and our guest today?

Sure. Yeah. We have a special guest. Special guest, Stefan Thiel. Hi, Stefan.

Hi, Stefan.

Hi, everyone. Thanks for having me. Hi.

Pleasure as always. And Stefan was part of the design team. Some of you for Emoto, for the project, we'll talk about in a minute. He also did his bachelor's in Potsdam, so that's where we met. And I guess some of you might know him from the visualizing Shakespeare project, which was his degree project and where.

Yeah, in 2010.

Yeah. And he did some really interesting text analysis on Shakespeare texts and made visualizations out of that. Big posters, interactives and. Yeah, that's correct. Yeah. And now he's part of studio NAND, the best computational design company in the world, as everybody knows, and was part of the.

Probably not, but. Yeah, together with Jonas Law and Stefan Fiedler. And Stefan.

See, that's what I'm saying.

You finally teamed up, and it's really exciting how it goes. Of course. Exciting time behind us with Emoto and stuff. Yeah.

This is adding up to all the fees we get every time we have an episode.

So. Yeah, Emoto. Yeah, it's been a big thing for us. We've been working on that. I mean, it started like half last year in summer, like, to get the project going slowly and get some funds and so on. And since March, we knew we would be. We would get some funding and started working on it. So it's. It's been one of the big projects this year for all of us, I guess.

Yeah. Really exciting one. And long term, definitely.

And lots of a multi headed hydra.

Which was also the fun thing about it, to bring together so many components and so many pieces. But let's talk about them in detail.

Which was also the fun thing about it, to bring together so many components and so many pieces. But let's talk about them in detail.

Yeah. From the outside, I must admit, it looks huge.

I mean, whenever you think you understood what it is, then there's an extra subpage somewhere. But let's try. Try and characterize what we did there. So the basic idea was. So there were the Olympic Games in London. We knew that before because they were sort of announced. And Drew Hammond, he's from Manchester, and he's running the future everything festival, and he's quite active, generally in digital media, organizing lots of different activities. And he had this idea, okay, we have the games. We need to do something, you know, to visualize the online response to the games, you know, not just the. Not just the tv feed where you see the athletes, and not just the live experience in the stadium, but what are all the people on the Internet, you know, saying as the events happen.

Yeah.

So that was the starting point, really. And I think that the. At the beginning, the idea was really to capture the whole stadium atmosphere and the excitement of viewing, like, sports together. I think as the project progressed, we moved a bit away from that, towards a more analytical view and a more, say, more distant view, maybe, but I guess that was the starting point.

So, basically, the initial commission was like, research for what might be interesting in terms of data. What kinds of data can you get from the Olympics and stuff like that. And also we tried to be really open about our focus there and played around with different scenarios. We were really open in the beginning, weren't we, Moritz?

Yeah. We also looked at processing tv streams, doing image analysis on tv images. Or we thought we could tune in all the radio stations of the world at once, you know?

Yeah. Mix up and mash up all the tv streams and everything together, working with subtitles, or maybe sensors on location in the stadium and stuff like that.

Okay, so you didn't start with the idea of analyzing Twitter feeds directly?

No, not directly, no.

Okay.

We knew it would be there as an option, and it is really challenging and exciting to work with this data. And this is why we have headed in this direction, basically. But, yeah, we try to be really open and consider all other possibilities as well.

Yeah. Okay, cool.

And, I mean, a big inspiration has, of course, always been, we feel fine. From Jonathan Harris. I mean, how could it not be? Right? But that was always us in the back of our minds. Okay, let's do. We feel fine. A bit cooler. Which is like a bad premise for any project.

So who is behind the project? I mean, practically speaking, who's been working on the. On the project itself, coding, designing, prototyping, analyzing all this stuff?

Yeah, it's been a big team, so I can give you sort of the chronology a bit. So, as I said, drew had the original idea, Drew Hammond. So then he approached me if we would work together on sort of fleshing out that idea and developing a concept that would be able to be funded or be handed in for a funding application. And then I said, yeah, that sounds really great, but I'll need some help here. And so then I approached the guys from studio NAND to work already on that first concept, and so we did that together.

And this started in August already.

That was last year, augury. Oh, man, that's crazy. Yeah, yeah, yeah. And then we handed that in for different cultural funding pots, so to speak. And we also tried to do the official route and place it on the official London 2012 website and talk to Lowcock, the organizing committee, and we did even a pitch there. But luckily, I must say, we did not succeed. I'm really happy that we didn't get the official commission because this one would have been very hard to produce. You know, like, support all the different browsers, do it in all, control all the contents at any time. You know, it would have been a very, let's say, very restricted project from the creative side.

Yeah, sure, sure. Okay.

Yeah. And then in March, it became clear that our arts council funding would come through. And then we knew our summer was ruined. That was everybody's first thought, like, oh, my God, there goes my summer vacation. We were, of course, also happy.

So, practically speaking, you and guys from NAND Studio have been working on the.

Project in practice, in terms of design. In terms of design final concepts and stuff like this. Yeah, we have been developing this mainly in close collaboration with Drew. But, yeah, I mean, the design work and stuff like that is all Moritz and Studio 19.

Fantastic.

And then we had, of course, we had a big organization team around it, people working on pr, people working on producers, using the exhibition, you know, all this extra work that you don't really think of in the beginning, but there's a lot of nitty gritty stuff to be taken care of.

Okay.

And Gerrit Kaiser was our backend guy, so he took care of doing all the backend work for retrieving, analyzing and providing the tweets to the front end, which was quite a task, I have to say.

Yeah. And we must also mention at this point that we have been supported by lexalytics for the data analysis, for the sentiment analysis, which is also we should, we should mention it as a team because they really gave a great support and helped really a lot with their software. They're providing it to us without any funds and stuff like that. So really lexolytics should be considered part of the team because their software component provides a sentiment analysis, which we found in the research is the best on the market. So.

Okay, who was that?

Lexalytics, it's a company based in the US. And they really did, this is what they do. They do sentiment analysis and provide a software package and also for entity retrieving and stuff like that. So they have a really comprehensive software package and we are using the sentiment analysis component from that.

Okay, fantastic. Okay, so before we delve into the technicalities of the project, do you guys want to briefly describe what you have on the Emoto website in case people never saw it, or just to refresh their mind and, yeah, briefly mention what's the final outcome of the project. And of course mention the URL in case people want to go there.

Okay, fantastic. Okay, so before we delve into the technicalities of the project, do you guys want to briefly describe what you have on the Emoto website in case people never saw it, or just to refresh their mind and, yeah, briefly mention what's the final outcome of the project. And of course mention the URL in case people want to go there.

Yeah, I mean, what you finally see on the website www.emoto2012.org is a visualization which is mainly consisted of two, or which mainly consists of two components, which is one is the origami style figures that we see as one core. This is how we call them. We don't have a specific name for them, but they look like origamis and nice little origamis. And this is how we call them. And this is what we use to visualize topics in a real time manner, but also over time. So we have triangular shapes which are giving a feedback on the overall sentiment distribution for a specific topic around the Olympics. And the topic could be an athlete, a discipline, or even other social topics such as the traffic situation in London or the empty seats at some of the venues, which were a bit of a negatively discussed topic during the Olympics and stuff like that. So for each one of these topics we're having little composition of some colored triangles, and triangles in the top represent positive emotions and distribution of the sentiment levels that we have determined. And triangles on the bottom are the negative ones. So you can easily sort of like a thumbs up, thumbs down indicator, see how positively or negatively is a topic discussed at the moment. And then there is also a real time message stream which really focuses on seeing Twitter messages in real time, flying by. And this is supposed to be a compliment, like, next to watching tv, you know, we thought it might be interesting, or we had Moritz did one of the early prototypes there, and it was really. We knew. It's really so direct, so immediate. You see the responses just right as they happen. And we thought it would be nice to just have that as a compliment to watching tv, for instance, or something like that.

Did you mention people sitting in front of their tv and their iPad on?

Yeah, exactly.

Second screen is the word of the year, probably.

Yeah, definitely. I mean, more. It's a way. Right? We imagine it to be like this and hope it will be like this in a way. Yeah, absolutely.

But I must say, I found it's very challenging to watch sports and read at the same time.

Yeah, definitely.

So I think it works nicely for American football where you have breaks. No, but it's really difficult for sports that have continuous action going on. So I tried it a few times, and it. It's not so relaxing.

I think it depends very much on how much you want to focus on the game. Right?

Yeah. And a lot of little design decisions, like how you work with attention and how you work with changes. And so there's. It's very interesting, but it's also challenging to do a good second screen application.

Probably one good thing to mention as an effect is we did some first studies around the Euro soccer Cup, and this was actually the first time that this kind of prototype of the stream was running with live data. And when the. When for one game, I think it was Germany against Poland or some other. I mean, it definitely was Poland, which scored the 10 to the first goal, and the excitement was immediately visible in the real time stream. And this was like a clear indicator. It's more like, bae, there's something happening. It's like a warning light, basically. And this is what we really liked about it.

Okay.

It's sort of magic. I mean, I also had that with Usain Bolt, you know, when he ran the 100 meters in 963, it was in that very millisecond, the first tweet came, and I was like, wow, how did that happen?

All news agencies are behind.

So the tweet had the same delay as the tv picture, you know, and it was like, wow. Yeah, it's sort of magic.

News agencies take up to one or two minutes to put that on their website. So this was really what was exciting about it.

Yeah.

So you actually touched upon something that I wanted to ask you later, but so I was wondering, by looking at your project, how did you actually test your system before going online? Because this looks like the kind of things that could really blow in your hands and the kind of thing that. And you would never have enough confidence without some kind of real data. Right.

Yeah, that was, it was a huge challenge. I mean, we did collect test data from the golf masters as well as the soccer Eurograph, which is not exactly.

The same page, which has bolt running.

Exactly. And the other problem is, it's single event sports, you know, where one thing is happening at a time at the games. All kinds of things are happening at the same time, you know, so, yeah, so we did at least have some idea of, you know, what sports data looks like Twitter wise, but we had no idea what Olympics data looks like until the game started. And, yeah, the first few days we had to fix and tweak a lot of little things just because before, it was very difficult to have a realistic test setup. And I read the BBC, they actually wrote a full simulation of the games to be able to test their programs. So they had this sort of artificial games feed that they would generate so they could test their real time components.

Okay. Okay, so basically, you had this kind of base jumping effect where you just threw yourself.

Yeah. You just have to jump and. Yeah. Make the best assumptions and roll with it. Yeah. But it was challenging, definitely. And that's a big issue with real time data that it's hard to test. And you have, in my experience, I mean, now, in hindsight, I. We should have written a full simulation that continuously runs so we always have something to test against.

So you didn't try to create some simulated data out of it?

Oh, we did, yeah. So we had sort of this replay of this one soccer match, you know, being fed in over and over, or I think the golf masters, we used more in static analysis, but it wasn't that close to the real thing as it could have been.

Okay.

Yeah, yeah.

Good, good. Wow. Yeah. Look scary.

And it was an adventure, definitely.

I mean, we are, in a sense, lucky that it was an art project and not like, for, like, top notch client. That would kill us when the site is offline for five minutes. No, but then it's really tough, too. So I have huge respect.

Right. I mean, it's. It's quite normal that you get some. Once you go live, you will normally almost always have some little problems at normal. So I was curious about. I wanted to start. I would like to know something about the data processing and analytics you have behind before going into the visualization part. So how do you actually get this data from Twitter? I guess this is a small percentage of the old stream, right?

Right. I mean, it's. It's quite normal that you get some. Once you go live, you will normally almost always have some little problems at normal. So I was curious about. I wanted to start. I would like to know something about the data processing and analytics you have behind before going into the visualization part. So how do you actually get this data from Twitter? I guess this is a small percentage of the old stream, right?

Yes, yes.

How do you access this data and what kind of processing you do on top of it? I was really curious to hear about that.

Yeah, I mean, in the end, we've turned out to be using the public Twitter streaming API, which is only a 1% subset. And we were initially trying to get a higher percentage of the entire Twitter stream to have a more significant coverage in terms of statistics, and also for not so frequently discussed topics, to get more tweets from them, basically. So to have better, a better overview over the topics.

So that's what they call the fire hose?

No, that's not what they call the fire hose. No, that's what they call the garden hose, I guess.

Okay.

But it turns out in hindsight, this garden hose was really kind of enough.

Yeah.

Okay. I mean, it's, of course it's more, you increase the resolution or density of the representation in the data, but I think the 1% is actually, for a project like this is actually kind of enough. And it was already providing lots of technical challenges because we have an entire infrastructure already for this 1% host running on Amazon cloud services. So we have collected more. It's twelve and a half million tweets.

Something like that.

Yeah, something like that. Wow.

So these are all tweets that are clearly related to the games in English with a sentiment score. So they have some sort of emotional word in them.

So, yeah. And what we are doing is we have several machines running on node js, and they receive the tweets. And between each machine, each machine is responsible for one specific part of the project. So one is ingesting the tweets from Twitter, the other one is pushing topics and data to clients which are connected to the front end. And in between there is stuff like archiving, storing away tweets, and also, of course, the sentiment analysis project. And we communicate with these machines, communicate over Redis, which is a high performance storing engine. Basically, each machine just gets the data in, pushes it into Redis, and the next machine is notified when something has been pushed to Redis and keeps processing the tweets and passing it on to the next one. So we have a sort of buffered, or a chain of computers.

The Internet is a series of tubes.

Exactly.

It was really this sequence of pipes that worked out really well, because then each component can do sort of one thing properly and then pass on the result to the next component. It's really? With like. Yeah. Unix pipes or so. Yeah.

Okay. So you actually filter out the messages making sure that you get only those that talk about the Olympics and have some sentiment.

Yes.

Maybe you have more parameters than that. I don't know.

Yeah, I mean, we are doing an initial, very general query about the Olympics. So we asked Twitter for all the tweets regarding which mention London 2012 Olympics, and this is about it.

So you have official Twitter accounts? Yeah, yeah.

And so, of course, then some specific Twitter accounts.

Exactly.

And so we try to get the entire group of messages, or as many messages as possible, which could be related to the Olympics, and then we further process them. So first of all, we distribute them in separate channels where we just look for specific keywords which we try to match the topics against. So, for instance, for swimming, we try to, we have several keywords like disciplines like athletes and stuff like that. And this is how we distribute messages in different channels. And then we do the sentiment analysis and further reduce the amount there by just kicking out everything message which had a sentiment score of zero. So basically it was neutral.

So sentiment score zero is no sentiment or neutral sentiment.

It can be both. It can be either no sentiment detected or. Yeah, it basically says no sentiment detected. Neither a positive or a negative word was found. So that sentiment analysis is based on looking for words that have a positive or negative connotation or tone, and then they have scores. And the software will also detect negation. So if you say not bad, it will realize that that's actually quite good.

Okay.

And so, but if none of these words sort of that could have an emotional meaning, are in the tweet, then it won't show up in our system because we said we want to focus on the emotional, you know, response and like if people cheer or curse and get the extremes. So we were personally quite interested in the extremes, not so much in the very obvious neutral reporting that you get from any news source. Right, sure.

And this part has been done using this software that you mentioned before, lexalytics. Okay, great.

Yeah, we should mention them every five minutes because they, like, we asked them like one day and like half day later they wrote, yeah, sure, you can use our software. It's fantastic. We're happy to support you now. It was really. And it saved our asses basically.

Yeah. And I guess doing these things from scratch would be a nightmare.

No way, no way, no way. You wouldn't at all reach the quality. So we did some studies with open source tools, but you wouldn't reach the quality that lexalytics provides. And this was key to the project. So if the sentiment analysis is not good, then yeah, yeah. What should we visualize then?

No way, no way, no way. You wouldn't at all reach the quality. So we did some studies with open source tools, but you wouldn't reach the quality that lexalytics provides. And this was key to the project. So if the sentiment analysis is not good, then yeah, yeah. What should we visualize then?

But one thing I didn't understand from the project, do you have also some kind of topic detection algorithm there? Because you mentioned you have one of the visualizations is about topic. Right. I was wondering whether you automatically detect interesting topics there and how you do it.

No, it's supervised. So we have an expectation of what interesting topics were or could be, and then we look for indications of that topic, you know, and this is much easier and for something, for the games, also feasible because there is a limited set of things people are talking about. Right. And trying to detect yourself, what are the trending topics or the hot topics. So it might look very similar in the end result, but it's much more robust to say, okay, we are looking for disciplines, we're looking for the, let's say, top 50 most popular athletes, you know, and maybe some things that we during, as the games progress, realize are interesting topics. So we had a simple admin interface where we could easily make up new topics.

Right.

Oh, so you could manually actually add new topics as the.

Yeah, they had a label and a regular expression, and we could enter like a new topic and have a regular expression. And whenever that expression matches a tweet, it will be counted for that topic.

Okay.

Yeah, that was quite nice. Yeah, yeah.

Well, some data driven journalism style, which the entire project is a lot about as well, because Moritz has been done also, like really nice studies and explorations during the games, just putting data in Tableau and extracting the courses of specific topics and stuff like that. So this was a large part, I would say, also to explore a bit how data gathering and analysis and visualization can be used to just get an understanding of such an event in a different kind of way.

These different speeds are interesting, I think, because the web platform was very real time, like on the second, you know, and ever changing also. So, you know, it's always new. And then we had a slower pace in the, in the blog and in the sort of data journalistic parts where we would look every three or four days.

Yeah, sure.

Okay. What were the bigger developments? What were people talking about? Can we make some analysis on that and maybe contribute a bit from the data side, you know?

Yeah, yeah, yeah.

And then we had as the third part, the data sculpture, where we conserve all the tweets that we collected in one big 3d object. And so this is the slowest and.

It's conceived to stay there forever.

Exactly. It has this preserved forever. Feeling. Exactly.

Yeah. In a thousand years, they will find it somewhere.

Yes. We always liked that idea of the.

Rosetta stone in the art of London. Yeah, yeah. But I think that's a recovering topic. The idea, I think we've been discussing that during our last episode as well. The idea of really carefully thinking what's the best mix between what the machine can do and what the human can do. And, I don't know, this is another case when instead of pretending the machine to come up with interesting topic, you can use the machine just to have an overview of the game and then carefully, manually pick some topics and put them into the system. I think it's really fascinating. Again, once again, I think it's another example of interesting interactions between what the machine can do and what the human can do and how they can cooperate. I think that's really, really interesting here. Okay. Do we guys want to move? Do you guys want to move to visualization and talk about the design and the development of your visualizations?

Rosetta stone in the art of London. Yeah, yeah. But I think that's a recovering topic. The idea, I think we've been discussing that during our last episode as well. The idea of really carefully thinking what's the best mix between what the machine can do and what the human can do. And, I don't know, this is another case when instead of pretending the machine to come up with interesting topic, you can use the machine just to have an overview of the game and then carefully, manually pick some topics and put them into the system. I think it's really fascinating. Again, once again, I think it's another example of interesting interactions between what the machine can do and what the human can do and how they can cooperate. I think that's really, really interesting here. Okay. Do we guys want to move? Do you guys want to move to visualization and talk about the design and the development of your visualizations?

Sure. Yeah.

So if I understand well, the main things I've seen on the website is this origami visualization, then the real time tweets, then you have in the blog what you call the centigraph, which I really like as a term, and the heat map. And of course, you have the data sculpture. I hope I didn't miss anything. So do you briefly want to talk about how you went about, I don't know, prototyping this stuff, designing this stuff, conceptualizing, actually, this stuff. I'm sure you went through a painful process, as usual.

So if I understand well, the main things I've seen on the website is this origami visualization, then the real time tweets, then you have in the blog what you call the centigraph, which I really like as a term, and the heat map. And of course, you have the data sculpture. I hope I didn't miss anything. So do you briefly want to talk about how you went about, I don't know, prototyping this stuff, designing this stuff, conceptualizing, actually, this stuff. I'm sure you went through a painful process, as usual.

Yeah.

I mean, how did you come up with these ideas? I'm really curious about the origami thing. Sure.

Stefan, you want to start? Yeah, yeah, I can.

I mean, we initially. Excuse me. Of course, we initially started with, like, lots of explorations, you know, being inspired by different other projects that we find really interesting, which capture a lot of the dynamicism of the games, of sport events in general. So we did lots of experiments in the beginning really quickly, prototyping stuff with particles, for instance, and sports motion capturing recordings. So we initially thought, wouldn't it be nice if we focus on athletes and disciplines? Wouldn't it be nice to have figures which are built from particles representing a tweet of each, stuff like that, to have a really compelling, complex image, for instance, and also some dynamic animation and stuff like that? Then again, this looked really interesting. It's always interesting from an aesthetic perspective or from the visual impact, just. But I mean, then we also entered or started questioning these approaches again, in terms of, is it readable? Will people understand it? Is it meaningful? Is it actually useful as a sort of visualization? And this is how we then played a bit more around in parallel. We played more around with the data as well. And the initial key moment, Moritz, I think you would agree, was when we finally saw the, the first version of the sentigraph, it just points plotted on an x y axis. So you have the sentiment of a tweet on the y axis and the time basically on the axis. Pretty straightforward. But then we sat together and played around a bit and more extended little multiples of different topics and entities. And then we started seeing there is something hidden in there with just doing small multiples with this sentiment plots, basically. And then the topics and their discussions start to shape out from the data. And this is where we started to like, okay, how can we use this basic principle and have one indicator, one visual element which indicates the current state of the topic? And then we combine the two things, basically get that information across that we found there.

I mean, the other thing we realized, because in the beginning we had these particles forms, right. That would form like a cloud of something, you know? But then we realized, okay, the topics we're talking about, you know, there's such huge differences in tweet volume from, you know, between time points, but also across topics that whenever you do something where one tweet is represented by one particle, you're in trouble. You sort of adjust the number of particles so it looks better, or 99% of the time, you have too many or too few particles. And with particles, it's really important not to have too many or too few.

Yeah.

And that's, it's a big conceptual issue for these types of things.

Yeah, yeah, sure.

And we wanted to be sort of truthful to that, you know, that we wanted that each view of the visualization works regardless of how many tweets we have. Right.

Yeah.

And so then we realized, okay, we need to do something. We need to move it up one level, zoom out one level, and work with aggregates a bit more and with coherent shapes, you know, that stand for a whole set of tweets or something like that.

Yeah, that's really interesting to hear. I was curious to hear the story behind that because very honestly, the same, the first time I saw the visualizations on the website, I liked the design, but my first reaction was like, oh, I expected to have more data density here. There's not enough information. It's too aggregated. But at the same time, I fully understand what's the challenge here? And now that you mentioned the fact that, of course, in an event like that, you might have really high spikes of something and then very low volume and managing to have, I mean, having a visualization where you have single data points for every event or tweet or whatever is going to be a huge mess.

Right, exactly. And very hard to handle. And we wanted to make sure that every click is sort of interesting and has, you know, delivers enough information to be meaningful. And that's why, for instance, you cannot just look for enthusiastic tweets about a topic on day two, because it maybe it was only one or two, and then, you know, you're in that dead end of having filtered down your data too much and so forth. So that's why we kept it on this very general level, I guess.

And how did you come up with the origami? And on a side note, I would like to have a comment from Stephen Few about Yorick.

I was skeptical, too. I was skeptical, too.

Compared to a bar chart?

No. No. So I think Jonas can take credit for the origami principle, part of the studio NAND team. And I must say, he really, he convinced me there because in the beginning, I was skeptical. So I toyed a lot with different triangles of different sizes and tried to have sort of loose arrangements because I knew that it could work, that if you have independent triangles and they have different sizes, you know, that together could make a nice profile. But I never had a good way of arranging them. So in my prototypes, they were always, like, flying around fairly randomly, and he was always like, no, we need some structure here. We need some structure here. And then he came up with that folding principle, but only in illustrated. I was like, wow. But will that work for any combination of values? You know, what if they fold back into the center? Or what if they fold, you know, to the left side of the screen and don't go anywhere? And I was always like, oh, can we really do that? And then, yeah, a few weeks before we launched, it was clear, okay, now we need to decide. And he came up with a really compelling, like, proposal how to make the logo design integrate with the visualization, you know, and the colors. And I was like, okay, we have to do it.

Let's do it.

And luckily, it worked out. But.

It's a similar topic to that, wasn't it? Weren't there? Yeah, I mean, we had, like, we initially started with a two color scheme, which was really nice. I liked it, really in terms of, like, clarity. I mean, but then there was this at some point, we weren't sure anymore what should be positive colors and what should be negative colors.

Everybody had a different, I think we.

Chose a color scale, a pinkish, reddish and bluish color scheme, blue for. I think initially blue was positive.

It wasn't positive.

Negative.

Yeah.

Okay.

And then at some point, we thought, like, okay, let's switch it and make. Does it make more sense? And then we were totally confused, like, what is actually? What is actually, because we had also individual, like, individually different assumptions on what should be a positive and what should be a negative color. So we reached out to some friends and did a little private, short poll, I would say. And the result of that was that what most of the team was thinking, that the reddish, yellowish, vibrant colors should be considered positive versus the blue one. One's negative, was actually a good assumption, but still, it was a surprising amount of votes for also the blue color scales being positive colors. So, of course, it's highly subjective.

That's really interesting.

But at some point, we turned colorblind in the process, and we had to reach out to people and ask them and to be really sure, again, like, this is the real color that we've met. And also doing some more, greater research around different color schemes, making it a bit more, opening it up a little bit more. So it's also more interesting and allows more fine grained feedback.

Yeah, but it's really interesting because also we launched the project and, like, the second Twitter commentary is, wow, that's awesome. But I would revert the colors.

Yeah, don't go there.

We have discussed it. Yeah, it's interesting. It would be interesting to hear if there's any research. So I didn't find any, but I didn't look that deeply.

Did you think about crowdsourcing that? Like, I don't know, trying to have a little test on. We did it on small Mechanical Turk or anything.

We did a small one, but then again, I mean, it would have been too much to do it, like, really valid in a valid way, you know, reaching the necessary amount of people and also making sure the, the population, as I would say, on mechanical Turk, for instance, is the right one and stuff like that.

But I think I might be wrong. I think Collin Ware, in his book, briefly discusses the relationship between, or the semantic interpretation of colors. There should be something like that. Did you try to look into that?

Yeah. Yunus has also looked at some of them. Yeah, there were lots of papers that we read about different types of colors and their emotional interpretation and stuff like that. And this is basically how we also came up with the color scheme. We just changed it slightly to not be so extreme, reduce the browns a little bit, which is the slightly positive ones, and not making the, I mean.

That was the challenge that we wanted to have a graduation within, you know, the positive and the negative. I think if it's just about pick a happy color and pick a sad color, I think that's easy. Yes, but it's hard.

But if you want to find the colors.

Yeah, that has plus, that goes from plus six to minus six. That was our challenge, and that was, it was tougher than we thought.

Really.

Yeah. It's really interesting how much work can be behind just selecting some colors.

Yeah, yeah, that's for sure.

But now we're happy with it, I think. I mean, also, in hindsight, I say I think we made the right call. But it was. Yeah, it changed a lot and was a lot of discussion.

I have to say that for me, it was very natural.

Yeah, that's good.

Yeah, that's good. Yeah.

Good to hear.

Good. And what else, what about the real time tweets? How did you come up with the idea of having these moving small windows, and then when you click on them, you can fix some of them, it looks like. I'm sure you spend some time thinking about it, especially the interaction part, how to interact with this, and what's the main purpose of this visualization and how not to overwhelm people. I think that. I'm sure you've been discussing a lot on that, or maybe not.

Good. And what else, what about the real time tweets? How did you come up with the idea of having these moving small windows, and then when you click on them, you can fix some of them, it looks like. I'm sure you spend some time thinking about it, especially the interaction part, how to interact with this, and what's the main purpose of this visualization and how not to overwhelm people. I think that. I'm sure you've been discussing a lot on that, or maybe not.

It came a bit from my experiences with the revision visit project. And, I mean, it's a project where you can type in the search query, and then we'll show all the tweets from the last few days for that. But it will make a fixed arrangement or fairly fixed. I mean, it changes a bit over time, but it's fairly stable. And conceptually, we thought, because the website will play this real time role in the bigger context of the project, we wanted to have something that really reflects this constant flow of messages. Right. And emphasize that real time character. And so we came up fairly early with this idea that there's a constant stream of things flying by. That was our code name for that view, things flying by. And I played a bit with different ways to do that. And at one point I came up with that principle that big things are more important, which is fairly natural, but then they also flow more slowly. So we have, it's like an inverse parallax, where the things that are close move slowly but the things that are far away move fast, which is sort of strange.

Makes sense.

No, it doesn't. But that's the fun part. I mean, it doesn't make sense if you. If you try to see it as a perspective thing, you know, like if you look out of the train window, it's the other way around. Yeah, but if you. If you see it as things floating on a river, it makes a whole lot of sense. Right. Or things like that. So.

And the big ones are those that are retweeted the most?

Retweeted a lot. Exactly.

Yeah. I mean, the stream constantly tries to dynamically choose the tweets which should be bigger and should be smaller, because then, because, again, we have that problem that we could have a high volume stream for a really high volume topic at the moment, but also there could be a stream which just has five tweets per minute or something. So the stream currently, like, always calculates some on the fly statistics about what is the actual set. I am showing the set of tweets, and then tries to prioritize on the tweets which were more retweeted, but also which come from authors with more followers and stuff like that. So it's a mixture of the audience of the tweet and the amount of retweets it has.

Okay.

Okay. So again, yeah. It's through a really complicated logic to how to sample the tweets and to pick the ones to display to make sure each of the views on the site is interesting, you know, because we.

Knew there was a sweet spot for the stream as well, you know? And this is why we can define or have defined a certain number of tweets for each size to be shown in an ideal case. And the stream basically tries to buffer that and to always keep that time. So we have been experimenting with different screen sizes and then just saying, like, okay, for a really large screen, we need to, or it's ideal to show five large tweets and 30 really small ones. And on a smaller screen, it's obviously lower numbers. And so we've experimented with that and tried to always keep that number.

Okay.

Or these numbers.

So it's sort of, you have a sort of adaptive sampling rate according to how much stuff is coming in.

Yeah, you could say that, yeah. Or sort of a weighted sampling.

Yeah.

Yeah.

Okay.

It's interesting because that's sort of in between data visualization and interface design. You know, in interface design, you will want to pick views that are, like, have a nice level of depth and detail.

Yeah.

You know, and design that really well. And the visualization part, you want to represent what the underlying data stream actually is like. And here we are moving in between these two worlds, I guess.

How did you expect people to use this view? Did you have an idea or.

The stream view?

Yeah, the stream view. Sorry.

Yeah. As we said before, we initially hoped people would have it as a sort of second screen, just while doing other stuff. So the stream immediately reacts if there's something going on. And people get basically notified and then their attention changes from whatever they're doing to the visualization and they can start to get immersed into the visualization a bit more and read more tweets and play a lot more. So initially it was like, I wouldn't say ambient, but it's definitely like a second screen application we had in mind for this. And also, it just makes total sense to just really sit in front of. Of it and also read every tweet. It's really playful. And so it's both things.

You could open just briefly to see what are people talking about, about the games at the moment. And honestly, I think it was the best source for that. So also, with this prioritization of the more important tweets, I haven't seen that anywhere. The Guardian had a fairly simple, but quite similar second screen application, but I'm not sure how they selected it. Tweets there, that was a bit more intransparent. So I think just for looking like, okay, what's on at the games? Our application was really, really nice, you know, did you hear from the games.

And having the visualization next to it?

Yeah, we had to fix the site. No, we did that a few times.

But last time was the closing ceremony for the Paralympics on Sunday, which was really nice. And we just had a look at it and followed it a bit. Bit. And it was really like, yeah, it was really nice. It worked out really well, I think.

Okay, so, and I also wanted to talk about the centigraph. Apart from the fact that I loved the term, I really like the visualization tool. And, yeah, I really like this kind of neat, clean and very informative design at the same time. Time, it's really, really nice. And I think this is more part of the analysis that you do statically on the data afterwards. Right. Once the main events have taken place. And. Yeah, I don't know. Do you want to comment how you came up with this one first?

Sure. I mean, it's been, again, like a long road towards these final products. Somehow a lot of different influences played a role. I mean, so it was clear we want to look into the data as the games progress and already look for patterns and also see how we can maybe improve the visualization as things unfold, and also for getting a grip on the data. Stefan mentioned before that in the beginning, we made these simple scatter plots, like time on the horizontal axis and sentiment on, on the vertical axis. That helped us a lot in understanding just the texture of the data, and we found them always very attractive from an aesthetic point of view. But what was hard, because it's little dot clouds, more or less, was hard to see the overall development of things. So at some point, we started to introduce curves that would represent the average sentiment at any given point in time. And in the vertical positions, it's like a fever curve, more or less. But also the line thickness would give you a hint of the number of tweets. And these are like, turned out to be really strong charts because it's like as if somebody would press the pen harder, you know, because of that tension, you know, so it has this really calligraphic look suddenly. And so we always liked that and we never knew how to integrate it really, you know, or what the place of that would be, except our own back end sort of analysis. But then we realized we could make, as the games went along, make some dedicated analysis of individual topics like Usain Bold or Team GB or the little scandals and so on, and see how they are reflected in these Twitter sentiment averages and volumes.

And it was also a really important key component for the curation of the stories that we have presented in the installation.

That's true.

We needed to be able to explore what we have in terms of data and then just get an interesting perspective onto some different stories and to basically select them for what would be the key moments, the key stories of the Olympics that we would like to present to the audience. Like, look, this is where it works the best, and this is what we've learned and stuff like that. So for this, it was really a good tool and was, I think the team still wants to see that as a proper software for other things as well.

Oh, yeah.

There was the idea of websites being able to do that, you know, building interactive versions of this, of the centigraph and just getting in lots of data and seeing the graph for different data sets and stuff like that. Yeah, it's really a really nice piece. Definitely.

It was interesting also for the installation, so we had some ideas of what would make really strong stories. But if you want to have a good, let's say, data story, you know, you also have to see that the data is interesting as well. And for some topics, it worked out really well. And for others, we had a hunch like, oh, there might be a great story here, but the data didn't turn out to be that interesting.

So again, you had to manually hunt for interesting stories there.

Exactly. Yeah. And there were people on our team who were looking like, what are the papers reporting about? What are the big things being discussed in traditional media? And then we would try and go back and find out out how well is that reflected in our data? I mean, it's skewed. It's a certain way of analyzing things. Not everything that will be discussed in traditional media or in the general, let's say, discussion of the games will be reflected in the same way in our data set. And that was interesting to see. So, for instance, NBC, they got a bit of a bad rap for their reporting. But the problem was that wasn't reflected that well in our data because the term NBC is used in so many different contexts. For instance, if you just retweet something that NBC has written, you know, you're immediately sort of adding to that topic, NBC. And, and there, it was really hard to separate, let's say, the, the tweets talking about NBC reporting per se, or just retweeting NBC content, you know, you know, and stuff like that. So some stories reflect it really nicely and others not at all. Yeah. Yeah.

I really like it because it's very informative and you can, it's engaging and catchy at the same time. And you can, I mean, right now I am in front of me, the team Great Britain craze on Twitter. I would really like to dive into some of these stories and really learn more about, about these bubbles here or sudden negatives or stuff like that. It's really, really rich. I really like it. And I like the fact that you've been using the height and the width of the line at the same time. That's really nice. And it plays really, really well.

Gay for redundant encoding.

Yeah, yeah, it's good. It's really good.

Shall we talk about the data sculpture? Because we've been talking so long already about the online. I think it's an important part as well.

Yeah, well, the data sculpture is amazing. So yesterday we've been talking about that, me and you, Moritz, and. Yeah, please go ahead. It's really nice.

Yeah. I mean, was also, again, a really big part of the project as well and was really interesting because you see lots of sculptures that try to visualize data in some way, and it's really appealing because it deals with materials. You can use materials to reflect different things and stuff like that, and also how to deal with the third dimension just to visualize some certain things. And so it was really interesting to dive into all these aspects with the data sculpture. And I think the form we've come up with is really interesting because it's not just aesthetically looking good in a three dimensional way, but it's also kind of readable and to such an extent that we think we have thought about this, like, also offering it to, for instance, blind people to experience the dataset by touching and stuff like that. So the data sculpture is one object for each day, which we have milled with a manufacturer in Germany. And so it's basically a heat map, a three dimensional heat map, with each plate having a shorter side on the time axis and then the sentiment on a horizontal scale. And so you have, per sentiment, you have different bands that move along on the time axis and go up and down according to the volume of tweets we have for that specific sentiment at that specific time.

Like a little roller coaster, basically.

A little roller coaster. Exactly. So you have a little landscape. A roller coaster landscape. And on this object, we project different heat maps, as I would call them, for the individual stories. So you have the entire amount of tweets in the sculpture, represented in the sculpture, and then by projecting on top of the sculpture and just highlighting individual points per story, we sort of, like, you know, shape or highlight the individual important points for this specific story. So this is 16 pieces.

1717.

Sorry? Yeah, exactly. 17 plates, 17 objects. And so it's a three meter long table, and people can click through the different stories and see the heat maps and also have a little turning wheel with which they can move through time. And then we select specific tweets to display so they can see, like, what is this? Why is this peaked there?

Yeah. And that's how strong compared to the graphs, you know, where you say, Enrico, you would immediately like to dig in. Yeah. And see why the tweet look, you know, or why that dip in the data came about. And in this installation, you can actually read the most important tweet for each story, for any time point, and then that, or each hour of the day. And that was. It's so much fun to flip through all these tweets.

So where is the installation located?

It was located in Preston, in UK at the closing exhibition of we play. So it's the closing exhibition of the cultural Olympiad in the northwest, in UK. So there was a big party going on in Preston last weekend.

And now, Enrico, it's a bit like you. It has no real home.

You're going to have a new office soon, so if you want to send it.

You need something for your lobby?

Yeah.

No, no. We're looking for a good place to exhibit it permanently because I think it's a nice legacy for the games.

Absolutely.

It would be a pity if it would now sit in a box. I mean.

Yeah.

Yeah.

Wow.

Totally agree.

I could talk with some people at MoMA or.

Yeah. But I have to say, this 3d relief, you know, you might. It sounds. It might sound gimme tricky, but when you see that thing and this level of detail, you know, that you can put into a sculpture and then add this projection, I think it was. So I had the feeling, okay, we need to explore that more. You know, what is possible there because it's really interesting and it is worth the effort. You know, before, we were also not sure, is it worth going through all the effort, like, you know, preparing the 3d models and testing materials and calibrating the projection? You know, that's. There's a lot of technicalities there, but from the experience, I think it was worth the effort.

So definitely, I was wondering, when you are in front of it, do you feel like touching it?

Totally.

To be fully honest, yes.

Okay.

Would be good to explore that even further. It would be just lovely if we find a way to do that technically and then all these questions arise, like how do you deal with it on the interaction side, what to display and stuff like that.

You guys have to find a way to keep on doing this project in a way or another. It's fantastic. I would love to see it, actually.

It feels like the beginning, to be honest, not like the end.

Okay, cool. And, yeah, so, talking about the future, how do you go beyond this project? I'm sure, I mean, you've been doing a lot of work, a lot of cool stuff, a lot of thinking around it. And I'm sure you want to. I mean, I want you. I'm sure you want to reuse this stuff or I'm sure that some of the ideas that you've been generating during the project can actually be the beginning of new projects. Right. So do you briefly want to discuss that? How do you go beyond this Emoto itself?

Okay, cool. And, yeah, so, talking about the future, how do you go beyond this project? I'm sure, I mean, you've been doing a lot of work, a lot of cool stuff, a lot of thinking around it. And I'm sure you want to. I mean, I want you. I'm sure you want to reuse this stuff or I'm sure that some of the ideas that you've been generating during the project can actually be the beginning of new projects. Right. So do you briefly want to discuss that? How do you go beyond this Emoto itself?

Yeah, I mean, there were a few directions. I mean, we have to say at the moment we are really tired and mostly working on documenting and sort of wrapping up that phase now because it's been a long thing. And like, let's say the two weeks before the games, they were, they were tough. You know, we had to get this thing working. And.

How much coffee did you drink?

Oh, man.

Liters.

Gallons. Yeah. No, but, but there were a few directions. I mean, obviously everybody's interested in sentiment analysis. I think it's, you know, real time, and sentiment analysis is one of the things of the year, I guess, you know, definitely. And so that's, on the one hand, something that's really interesting, but also an area where you have to think about what you're doing. Right. But I think we collected some really good experiences with what works in real time, what works, and analysis wise as well. And I think we'd like to do another project in this direction. I mean, unfortunately, the us elections are now coming so soon. Otherwise they would be perfect. They would be perfect. But I think. Or something sports related. Why not? I mean, it works quite well.

It's applicable definitely to lots of topics and lots of situations.

So the next World Cup. Cup.

What I'm personally not sure yet about is if this big extra effort that goes into building a solid real time system, you know, that is really, really real time, pays off. But I'm also not sure what is sort of in between. But I had the feeling that the analysis wise, these things you would do over a couple of days and would involve a bit more manual analysis were more interesting content wise.

You mean that?

And I'm not sure yet how to. Yeah. What the sweet spot there is, but it's a lot of technical effort goes into building a really good real time system. And then visualization wise, it's always like, okay, you get the statistics for now, but what do they mean? You know, how do you refer back to that sort of more intermediate history and the long term history and stuff like that so that we didn't tackle that well, I guess. Yeah.

And with real time, you always have this problem of overwhelming people. Right. Or underwhelming that.

No, exactly. So there's a big challenge there. It's not something that works out of the box at all. I mean, not at all.

Yeah. And there is another thing I wanted to ask you. Are you planning to release this data openly or not?

I mean, we have a set of the tweets. We are also sharing it with a couple of scientists already that look into it now for long term things. I think if people address us, I guess we would be open to sharing it. Right, Stefan? I mean, we haven't really discussed the general case, but I mean, why not? It would be great if people do something with it.

I mean, I'm sure from my side, from the academic point of view, I'm sure there would be so many people around here that would be interested in analyzing that.

Yeah, good point.

I mean, I mean, it's a really odd topic in academia as well.

So we should mention, though, that we lost two nights of tweets. Yeah, that was one of the reflected.

In the sculptures as well. We see it as a bit of honesty that there's some flat areas, little small flat areas in the sculptures. Two of them. So it was really nice there as well because it shows that it's all real world and not made up, but.

This is how real data looks like came in.

Yeah, exactly.

So, yeah, if you can live with that, then just write us an email at infoemoto 2012. Dot, dot.

Fantastic. Okay. I think we can close it here, right? Unless you want to add anything, Moritz, Stefan.

Fantastic. Okay. I think we can close it here, right? Unless you want to add anything, Moritz, Stefan.

I guess I do. No, we could go on for hours, probably, because there were so many things, you know, in this project. But, yeah, again, we don't want to over or underwhelm and we don't want to welcome anybody.

Okay, well, thanks a lot for this fantastic project. It was a real pleasure to discuss this thing with you guys. It's amazing what you've done and, yeah, I'm curious to see what you will be doing next. Thanks, Stefan, for being a guest here and for being available with such a short notice.

Yeah, thanks. It was lovely. Thanks for having me.

Yeah, I hope I will have a chance to meet you somewhere around the world soon.

Yes, hopefully that's New York. I love New York.

Yeah, sure, sure.

We should work on that part. Yeah, definitely.

Cool.

Yeah, it was great having you, Stefan. Fantastic. Yeah.

Thank you.

Bye. Talk soon. And bye, everyone.

Bye, everyone. Ciao. Ciao.

Bye, everyone.

Ciao. Bye.