Determining What Video Conference Participants Are Typing from Watching Shoulder Movements

Determining What Video Conference Participants Are Typing from Watching Shoulder Movements

Accuracy isn’t great, but that it can be done at all is impressive.

Murtuza Jadiwala, a computer science professor heading the research project, said his team was able to identify the contents of texts by examining body movement of the participants. Specifically, they focused on the movement of their shoulders and arms to extrapolate the actions of their fingers as they typed.

Given the widespread use of high-resolution web cams during conference calls, Jadiwala was able to record and analyze slight pixel shifts around users’ shoulders to determine if they were moving left or right, forward or backward. He then created a software program that linked the movements to a list of commonly used words. He says the “text inference framework that uses the keystrokes detected from the video … predict[s] words that were most likely typed by the target user. We then comprehensively evaluate[d] both the keystroke/typing detection and text inference frameworks using data collected from a large number of participants.”

In a controlled setting, with specific chairs, keyboards and webcam, Jadiwala said he achieved an accuracy rate of 75 percent. However, in uncontrolled environments, accuracy dropped to only one out of every five words being correctly identified.

Other factors contribute to lower accuracy levels, he said, including whether long sleeve or short sleeve shirts were worn, and the length of a user’s hair. With long hair obstructing a clear view of the shoulders, accuracy plummeted.

Sidebar photo of Bruce Schneier by Joe MacInnis.

Seny Kamara on “Crypto for the People”

Seny Kamara on “Crypto for the People”

Seny Kamara gave an excellent keynote talk this year at the (online) CRYPTO Conference. He talked about solving real-world crypto problems for marginalized communities around the world, instead of crypto problems for governments and corporations. Well worth watching and listening to.

Sidebar photo of Bruce Schneier by Joe MacInnis.

Video: To make 1997’s Blade Runner, Westwood first had to create the universe

[embedded content]
Shot by Sean Dacanay and edited by Justin Wolfson. VFX by John Cappello. Click here for transcript. And if you want a close-up peek at the awesome Ladd-style logo Aurich cooked up for this video, you can get that right here.

Welcome back to “War Stories,” an ongoing video series where we get game designers to open up about development challenges that almost—but not quite—derailed their games. In this edition, we focus on a genre particularly near and dear to my dead, black Gen-X heart: the adventure game.

And not just any adventure game—we were lucky enough to be able to sit down with Louis Castle, co-founder of legendary game developer Westwood Studios. Castle’s hands were on some of the most famous titles of the 1990s, including Dune II, the Legend of Kyrandia series, and, most famously, the Command & Conquer franchise. But as wonderful as those games are—and as many hours as I spent lost in the woods of Kyrandia as a teenager—none of those mean as much to me as Westwood’s 1997 cinematic adventure game, Blade Runner.

You know the score, pal

Adventure games were one of the two ur-genres of true computer games (with the other being the arcade-style shooter), and as a child of the ’80s, adventure games were what got me into gaming. The genre reached its peak in the early to mid 1990s, with some of the best-remembered LucasArts and Sierra titles making their appearance thereabouts. But by the end of the decade the wheels had come off the cart, and it was clear that the genre was being eclipsed by the rise of the first-person shooter.

With the economic realities of the adventure game market in the mid-’90s becoming apparent, Castle’s pitch to create an adventure game set in the Blade Runner universe that would look and feel almost indistinguishable from the film itself might have seemed a little barmy. Worse, in order for the game to justify the amount of time and money required to meet that level of fidelity, the title wouldn’t just need to sell well—it would need to become one of the best-selling adventure games of all time (a difficult thing to do when your target genre has clearly aged past its prime).

Flipping the tortoise

Castle’s team faced a considerable number of challenges in bringing the cinematic world of Blade Runner to life using the technologies of the day, most of which stemmed from having to invent, from whole cloth, a way to seamlessly mesh their pre-rendered world with animated voxel characters (it turned out to be vastly more complicated than simply sticking a sprite in front of the background). Tackling this issue introduced an entire interconnected tapestry of difficult problems to solve, very few of which are faced by modern developers who can pick from ready-made game engines to license and use.

Fortunately for all of us, Westwood stuck with the challenge, even though finishing the game required more money than originally planned. The company built a title that isn’t just an homage or reflection of the original Blade Runner—it’s a legitimate companion to the movie, fleshing out the world in complementary ways and answering some key questions about the fictional 2019 described in the film. Although it’s somewhat difficult to play on modern PCs and suffers from an unfortunate lack of legitimate buying options, it remains a game worth finding and playing—the branching story makes for a high degree of replayability (something absent from a lot of adventure games), and it still looks and plays pretty darn good.

And that budgetary goal of needing to be the best-selling adventure game to date in order to make a profit? Blade Runner managed that, too.

I’ll tell you about my mother

This isn’t the only thing we were able to film with Castle, either—he was also kind enough to spend some additional hours with us talking about Command & Conquer, which had its own surprising number of challenges (like figuring out how to do hundreds of units’ worth of pathfinding on a minimum-spec machine). Castle was happy to go pretty deep into the weeds on the technical issues faced, and we were happy to let him. Stay tuned for that video in a week or two!

Mini 4WD is an electrifying race series for makers and tinkerers

[embedded content]
Video shot by Justin Wolfson and John Cappello, edited by Aulistar Mark. Click here for transcript.

We’re going to try something a little different this morning. Partially in response to several requests for more maker-focused videos and partially because my executive producer is head-over-heels in love with Pocket Circuit racing in Yakuza 0, we’re bringing you the first in what we hope to make into a series called “Mini Motors,” and it’s all about tiny cars going really fast.

RC racing in all its various forms has always been a maker-y kind of hobby, and Mini 4WD serves as an excellent genre example to start with. You take a 1:30-scale battery-powered car, spend days carefully and patiently tuning the crap out of it, and then you set it loose on a curving track as fast as its little wheels can make it go—up to 40 miles per hour (about 65km/h). The Mini 4WD that wins does so by a mixture of careful planning, careful engineering, and a big heaping of pure luck.

Must go faster

For this video, we spent time talking Mini 4WD with Randy Holt, owner of the HobbyTown store in Toms River, New Jersey. The biggest factor that sets Mini 4WD apart from other RC cars is that Mini 4WD cars are hands-off during the race—once the green flag waves, the cars are on their own. They zip around the track, steered by the cars’ built-in bumpers and rollers pushing against the track walls. Though the track appears to have multiple lanes in parallel, it’s actually a single lane that spirals around the circuit, connected by a jump-over. This ensures that all the Mini 4WDs on the track are all racing the same total distance (because otherwise the inner lanes would be shorter than the outer lanes).

Holt gives us a nice overview of Mini 4WD cars, the different race classes, and a bit of a primer on tuning and engineering. The big takeaway is that the sport is friendly to newcomers and easy to get into—you can spend $15 or so on the Tamiya Yaris shown in the video, which can be assembled and ready to race in about 45 minutes. It’s also a hobby that grows with you, and at the extreme end—if your interest runs that deep—you might find yourself adding carbon fiber parts and tweaking rollers and brakes by the millimeter to eke out faster lap times. Mini 4WD has something for all levels of racer, from casual to crazy.

A whole new world

This video has also been my introduction to 4WD Mini—and it’s a vast world with a long history, stretching back to the ’90s. Video editor Aulistar Mark is a veritable fountain of 4WD Mini trivia, and he passed this tidbit to me in email as the edit was being locked:

Mini 4WD is an interesting international phenomenon. One aspect we didn’t get into, is the 90s Anime Bakusō Kyōdai Let’s & Go!! which is bound to come up in the comments. Bakusō Kyōdai Let’s & Go!!, was localized in the US as the Saturday morning cartoon “Let’s & Go!!”. The series also had several licensed games for multiple platforms in the 90s, with a couple remasters released for mobile. This would be a precursor to the Yakuza series Mini 4WD mini-game. It’s great stuff for nostalgia, since the 90s cartoons were very much made like Bandai/Hasbro cartoons designed to sell toys.

If you guys like this pilot and like the series concept, we’d love to hear some ideas in the comments for additional racing circuit types to check out—please let us know!

Video: How Oddworld solved its narrative problems with mind control

[embedded content]
Video shot by Sean Dacanay, edited by Jeremy Smolik. Click here for transcript.

Some games entice you into playing them with loud marketing campaigns, sexualized cover art, or the promise of ludicrous over-the-top violence. But then there are games like Lorne Lanning’s Oddworld series—games that don’t lead with muscle- or bikini-clad heroes and defy easy categorization. Games like Oddworld tempt you into playing by promising a different kind of experience. There are guns and violence, sure, but the setting is strange, the plot is filled with gray, and the hero—well, Abe isn’t exactly sexy, or really even, you know, human.

But players who gave the original Oddworld a chance back in 1997 found themselves stumbling through a unique and fascinating world that was equal parts surprising and subversive, and the series has gone on to acquire legitimate cult-success status. With the approaching release of Oddworld: Soulstorm in 2020, we thought it was a good time to pay a visit to Lorne Lanning and his team at Oddworld Inhabitants, and talk about our favorite meat processing factory worker and his long journey from design notebook to screen.

“Write what you know,” they say…

We interviewed Lanning at the Emeryville, CA headquarters of Oddworld Inhabitants, the studio he co-founded with Sherry McKenna in 1994. For Oddworld fans, the office was a magical place, stuffed with the kind of memorabilia that amasses over more than two decades of game design. Lanning walked us through his journey to become a game creator, starting from his poor beginnings in what sounds like an unstable family. He got into video games because his father had a job at Coleco, and Lanning thought gaming would be a good way to meet girls.

Lanning’s ambitions weren’t aimed at the small screen—he had his eyes set on making movies. To pay the bills, he took a job at TRW Aerospace, where he worked on anti-missile defense systems (it was the 1980s, and Reagan’s Strategic Defense Initiative boondoggle was in full swing). His exposure to soul-crushing bureaucracy and supplier management formed the basis for many of the Brazil-seque ideas later presented in the Oddworld games.

But it’s the time Lanning spent at Rhythm and Hues Studios that had the biggest effect on Oddworld—at least the series’ collective look and feel. Working on visual effects set him on the path of visualizing game design in terms of cinema—not just how things were framed on screen, but also the discipline and budgeting style of the movie industry. When Lanning and McKenna (a fellow Rhythm and Hues alum) eventually started their own studio in the 90s, they approached their game and their character designs in the way Hollywood does. This obviously is de rigeur in 2019, but in 1995 when work on Oddworld started, it was most definitely not the industry norm.

The problem

When designing the first Oddworld game, Lanning and his team had to confront an annoying reality of game design—there are only so many ways to interact with the world in a side-scrolling action game, and a lot of those ways involve shooting stuff. And one of the immovable design goals of Oddworld was that protagonist Abe would go through the entire game without being armed—not because of any kind of political stance against guns, but because having Abe unarmed increases the character’s vulnerability in a world that’s already overwhelmingly hostile. A gun would provide an easy solution to many of the game’s problems, and where’s the fun in that?

It took some time to work out a solution, but Lanning the other designers decided that characters like Yoda don’t need guns to solve problems. They instead infused the game with a pastiche of mysticism drawn from a number of different sources, which gave Abe his secret weapon: the ability to possess other characters, including bad guys with guns. This let them then design in some puzzles involving shooting, which the player can solve by finding a bad guy, taking over his body, and having the bad guy shoot his way through the puzzle. To prevent the player from picking up the bad guy’s gun after the puzzle is solved, NPCs violently explode after possession.

An Odd(world) legacy

This video ended up being extremely long in the rough cut because Lanning gave us so much great interview material. We had to trim out quite a bit, but we’ll be producing an extended version if there’s enough interest in this video. There are several rabid Oddworld fans here at the Ars Orbiting HQ, and this video, like several others in the War Stories series, was a passion project with a lot of emotion invested in it (not to mention some custom voiceover lines performed by Lanning just for us!). We hope you enjoy watching it as much as we enjoyed making it.

DAM Or Web CMS? Part 2: Find Out Which DX Technology You Need For Workflow And Delivery

In the second part of our series, we take a look at workflow and delivery and where digital asset management and web content management systems excel. If you haven’t seen the first part of our analysis, make sure you check it out.

Do you need more help? Forrester clients can set up an inquiry with us.