The ingenious Posture Monitor app, from husband-and-wife team Jaka and Jasmine Jaksic, is a great example of using technology to change lives for the better. An award-winner in the Open Innovation category of the 2015 Intel® RealSense™ App Challenge, it addresses a problem plaguing an estimated 50 percent of US workers – back pain. Using an Intel® RealSense™ camera (F200) and several third-party software tools, the team turned the camera’s data stream into a handsome package of graphs and statistics – all while balancing power consumption and frame rate. In this article, you’ll learn how the Jaksics relied on a strong software engineering process to keep advancing toward their goal–and we’ll show you their frame-rate processing code sample, which illustrates how to minimize power consumption and still provide a smooth user experience.
Jaka Jaksic was co-founder and lead engineer for San Francisco-based startup Plumfare, a social gifting mobile app. Plumfare was acquired by Groupon in 2013, leaving Jaka on the lookout for another great opportunity. Jasmine is a long time product- and project-manager and currently works at Google. Combining their expertise, they started the company JTechLab. The couple decided to pursue a product related to posture problems—something they both have battled. The app has moved from prototype to production, during which time Jaka encountered and overcame a few interesting hurdles. The lessons learned include data conversion, power conservation, and integrating Intel RealSense technology with commercial software tools.
Posture Monitor continually monitors your posture and alerts you to potential problems using advanced graphics and statistics, packaged in an attractive interface.
Contest Deadline Spurs Rapid Advances
Like many who suffer back pain, Jaka had tried several products for posture correction—without success. “It’s not that difficult to sit straight,” he explained. “But it's difficult to do it all the time, because it requires constant attention. While you are focused on work, your posture and taking breaks are usually the last things on your mind.”
One day, Jaka got the revolutionary idea of using a 3D camera as a posture-tracking device, and, after just a little research, he landed on a solution with the Intel® RealSense™ technology. “I also noticed that there was a contest going on,” he said, “I got busy right away, and just made the deadline.” Successful applicants received an Intel RealSense camera (F200) and the complete Intel® RealSense™ SDK as encouragement to create the next great app.
After the Intel RealSense camera arrived, Jaka built his first working prototype in about two days, taking half that time to learn about Intel RealSense technology and set up the camera’s data pipeline, and the other half to build the first posture detection algorithm. Once the proof of concept was complete, he proceeded with further development and built the more polished app.
At this point they began some informal usability testing. “Usually, in software projects, it’s good to get as much feedback as possible early on, from all sorts of people,” he said. In this case, the amount of time was very limited by the project deadline and by the fact that both he and Jasmine had separate, full-time jobs. “As a general rule, the user interface (UI) is crucial,” Jaka explained. “Right after the technological proof of concept that verifies that the thing can be built, I would recommend focusing on the user experience.” That may mean a prototype tied directly to the UI, and some targeted questions:
- Do you understand what this UI does?
- Can you tell how to use the product?
- Do you find it attractive?
Only after positive responses to these questions, Jaka says, should the development process turn to creating functionality. “Never wait to create and test the UI until after you have a complete functionality, with considerable time and effort already sunk into development,” Jaka said. He pointed out that once your project is too far along, it’s very costly to go back and fix basic mistakes. “User feedback is the kind of thing that can always surprise you. We’re fairly experienced with application design, but still we’re just now finding things that, when we sit the users in front of the app, they say, ‘Whoa, what’s this?’”
It can be expensive—and time consuming—to get your app prototype into a UI lab, but the benefit for doing so is big savings down the road. In addition to asking friends and colleagues, one cheap and easy method of user testing that Jaka employed in the past is to go to a local coffee shop and give people $5 gift cards in exchange for their feedback. “People are usually happy to help, and you will learn a lot.”
Advice from an Expert App Designer
Jaka said that the demos provided by Intel are extremely useful—but he had a few words of caution. “Intel’s examples are technology demonstrations rather than starting points for building your own application, so you have to strip away all the unnecessary demo functionality,” he said.
For Posture Monitor, the camera’s data pipeline was the essence, and Jaka drilled down to exclusively focus there. Since the SDK didn’t stay centered on the user’s face at all times, Jaka used the raw data stream and processed it himself. He used the Intel RealSense camera to locate the user’s face, which he then converted to a rectangle. He could next approximate where the spine was by calculating the center of the subject’s torso. By noting where the pixels were and where the head was, he could continually calculate the true center of gravity. He noted that much of his approach will change when he adopts the Intel® RealSense™ SDK R5 version, which will support body-tracking using a new camera, the user-facing SR300. It will be available in Q2 2016.
Jaka also overcame limitations concerning infrared 3D camera physics. While skin reflects infrared light easily, the reflection is occasionally muddied by certain hairstyles and clothing choices. (Busy prints, dark colors and shiny fabrics may pose a problem; as could long, dark hair that obscures the torso.) From the depth-camera standpoint, certain combinations report that the user has no torso. “It’s like they’re invisible and just a floating head,” he said. There isn’t much you can do in such cases other than detect when it happens and suggest the user wear something else.
In order to work for everybody, Posture Monitor requires each user to complete a calibration sequence: they sit straight, demonstrating perfect posture, and then click “calibrate.” The application compares their posture at a given time to their ideal posture, and that’s how it assesses what’s good or bad.
The calibration sequence of Posture Monitor ensures that the system can identify key aspects of your body and thus track your posture.
The team has yet to use specialized medical knowledge or chiropractic experts, but Jaka says that day is coming. “We wanted the application to be able to detect when the user is slouching, and the current version does that really well. After we launch, we’re going to reach out to medical professionals and add some more specialized functionality.”
Minimizing Power Consumption
At full frame-rate, Intel RealSense applications are too CPU-intensive to be used in a background application. The obvious solutions are to only process every N-th frame or to have a fixed delay between processed frames. This is typically a good tradeoff when the user interface is not shown and responsiveness does not matter. But what if we want the best of both worlds: minimize power consumption and still provide a smooth user experience when required?
Jaka developed a frame-processing pipeline with a dynamic frame-rate, where the baseline frame-rate is low (for example, one frame every two seconds), and is elevated only when a visible control requires it. Using this technique, Posture Monitor uses less than two percent of the CPU when minimized–or when no real-time controls are shown–without any degradation of overall user experience. It’s a relatively simple and completely generic code pattern that’s easily applicable to almost any app.
Here is the sample code:
using System;
using System.Drawing;
using System.Threading;
using System.Windows;
using System.Windows.Controls;
namespace DynamicFramerateDemo
{
class CameraPipeline
{
public static readonly CameraPipeline Instance = new CameraPipeline();
// Baseline/longest frame delay (this is used
// unless a shorter delay is explicitly requested)
private const int BASELINE_FRAME_DELAY_MILLIS = 2000;
// Timer step / shortest frame delay
private const int TIMER_STEP_MILLIS = 100;
private PXCMSenseManager senseManager = null;
private Thread processingThread;
private int nextFrameDelayMillis = TIMER_STEP_MILLIS;
public int CapNextFrameDelay(int frameDelayMillis) {
// Make sure that processing of the next frame happens
// at least within the specified delay
nextFrameDelayMillis = Math.Min(nextFrameDelayMillis, frameDelayMillis);
return nextFrameDelayMillis;
}
public void Start() {
// Initialize SenseManager with streams and modules
this.senseManager = PXCMSenseManager.CreateInstance();
senseManager.EnableStream(PXCMCapture.StreamType.STREAM_TYPE_COLOR, 640, 480, 30);
senseManager.EnableStream(PXCMCapture.StreamType.STREAM_TYPE_DEPTH, 640, 480, 30);
senseManager.EnableFace();
senseManager.Init();
// Frame processing thread with dynamic frame rate
this.processingThread = new Thread(new ThreadStart(delegate {
while (processingThread != null) {
// Sleep in small increments until next frame is due
Thread.Sleep(TIMER_STEP_MILLIS);
nextFrameDelayMillis -= TIMER_STEP_MILLIS;
if (nextFrameDelayMillis > 0)
continue;
// Reset next frame delay to baseline long delay
nextFrameDelayMillis = BASELINE_FRAME_DELAY_MILLIS;
try {
if (senseManager.AcquireFrame(true, TIMER_STEP_MILLIS).IsSuccessful()) {
ProcessFrame(senseManager.QuerySample());
}
} finally {
senseManager.ReleaseFrame();
}
}
}));
processingThread.Start();
}
private void ProcessFrame(PXCMCapture.Sample sample) {
// [Do your frame processing and fire camera frame event]
}
}
// Sample control that sets its own required frame rate
class CameraViewControl : UserControl
{
// This event handler should get called by CameraPipeline.ProcessFrame
protected void HandleCameraFrameEvent(Bitmap depthBitmap) {
if (this.IsVisible && Application.Current.MainWindow.WindowState != WindowState.Minimized) {
// While the control is visible, cap the frame delay to
// 100ms to provide a smooth experience. When it is not
// visible, the frame rate automatically drops to baseline.
CameraPipeline.Instance.CapNextFrameDelay(100);
// [Update your control]
}
}
}
}
The Start() method initializes the Intel RealSense technology and starts a processing loop with a fixed delay (TIMER_STEP_MILLIS). This delay should be the lowest frame delay that your application will ever use (for example, 100 ms). In each loop iteration, this interval is subtracted from a countdown counter (nextFrameDelayMillis), and a frame is only acquired and processed when this counter reaches zero (0).
Initially and after every processed frame, the countdown timer is set to the baseline (longest) delay (BASELINE_FRAME_DELAY_MILLIS), (for example, 2000 ms). The next frame is processed only after this time, unless during this time any agent requests a lower value by calling CapNextFrameDelay. A lower delay / higher frame-rate is typically requested by visible user-interface controls (such as the CameraViewControl example), or by internal states that demand a higher frame rate. Each such agent can set the maximum acceptable frame delay and the lowest value will win; this way the frame rate always meets the most demanding agent. The beauty of this solution is that it is extremely efficient, very simple to implement, and only requires one additional line of code for each agent to set its required frame rate.
The Right Tools
The list of tools and technology that Jaka integrated with the Intel RealSense SDK to create Posture Monitor is impressive:
- Microsoft Visual Studio* 2013 (dev tools for Microsoft Windows* apps)
- Microsoft Blend* for Visual Studio 2013 (interface designer)
- .NET framework 4.5.1 (umbrella for packages, compilers, and runtime)
- AForge* framework (for image analysis)
- Accord* framework (for statistics)
- OxyPlot* (for charts and gauges)
- SQLite* (for storing posture data)
- Hardcodet Taskbar Notification (provides tray icon and popup menus)
Jaka said he used Microsoft Visual Studio and C# because they are industry standards for building Microsoft Windows* applications. He wanted to use tools that had the largest community behind them, with lots of third-party libraries available. He picked additional libraries for each particular need, trying many different products and picking the best ones.
Jaka didn’t write or use any plug-ins to get the Intel RealSense technology working with the application. He said that the SDK itself provides solid data structures that are standard and easy to use. “Sometimes you might have to use raw data,” he said. “We used the raw depth format with 16 bits per pixel, which is the most precise way of reading raw data.” He then wrote an algorithm to convert the data into a bitmap that had a higher contrast where it mattered. His converted bitmap focuses on the range where the person’s body is, and enhances contrast and accuracy around that range.
Posture Monitor integrates leading statistics and data-charting programs with the Intel RealSense SDK to produce an intriguing user interface full of helpful information.
To process the camera data, Jaka used Accord, a very extensive library for math and statistics. He liked that Accord also has some machine-learning algorithms and some image processing. The data had to be converted into a compatible format, but, once achieved, it was a great step forward. “Once you get the data into the right form, Accord can really help you,” Jaka said. “You don’t have to reinvent the wheel for statistical processing, object recognition, detecting things like shapes and curves—that type of stuff. It really makes things easier.”
Another tool, OxyPlot, is an open-source charting library that Jaka found to be very extensive and very flexible.
Avoid Technical Debt—No Sloppy Code!
Jaka has a general philosophy for development that has consistently brought him success. “Paying attention to code quality from the start pays dividends later on,” he said. So he starts by learning everything he needs before he starts coding. He’ll make a ‘throwaway’ prototype to serve as a proof-of-concept, which allows him to play with all the technologies and figure them out. At that point, he’s not focused on the quality of the code, because his goal is simply to learn. Then, he’ll discard that prototype and start over with a strong architecture, based on what he’s figured out.
At this point, Jaka is then ready to build high-quality components in a systematic fashion. The complexity of the code base always grows with time, so early decisions are critical. “You can’t afford to have any sloppiness in your code,” Jaka warned. “You want to clean up every bit as soon as you can, just to make sure that in time, it doesn’t become an unmanageable nightmare as technical debt piles on.”
Posture Monitor only works with a desktop or laptop PC, not mobile devices, because it needs a stationary camera. And right now it only works with Windows, because that’s what the Intel RealSense SDK currently supports. When the Intel RealSense SDK integrates with Apple MacBooks*, Jaka is ready to pursue that. And his ambitions don’t stop there—he’s interested in learning more about Intel RealSense technology working in some fashion with Linux*, too.
Jaka has also been thinking about building a specialized hardware device for posture monitoring, perhaps a product that would include an Intel® Atom™ processor. “I’m sure this work is going to take us in a very interesting direction, once we start interacting with the users and the medical community. We are looking forward to where the future takes us.”
A Revolution is Coming
From his perspective as a successful entrepreneur who has already struck gold once, Jaka believes that the Intel RealSense camera and SDK are reaching developers at a crucial time. He sees the global software market for desktop, mobile, and web apps as oversupplied. “Almost anything you can think of has already been built,” he believes. Now, with Intel RealSense technology, Jaka says it is much easier to innovate.
“This market is still fresh and unsaturated, so it’s a lot easier to come up with new and exciting ideas that aren’t already implemented,” he affirmed. “I think this is a really good time to start working with Intel RealSense technology. It’s a great time to dig in and get a head start, while the market is still growing.”
As one example of this, Jaka can envision combining Posture Monitor with IoT devices. “There are so many potential ideas right now that are just waiting to be built. Most of them, I’m sure, no one has even thought of yet, because the technology is so new. I think we’re in front of some really exciting times, both for developers and for consumers.”
Resources
Learn more about Posture Monitor at https://posturemonitor.org/
Download the Intel® RealSense™ SDK at https://software.intel.com/en-us/intel-realsense-sdk
Learn more about the 2015 Intel® RealSense™ App Challenge at https://software.intel.com/sites/campaigns/realsense-winners/details.html?id=22