Simple RGB Streaming with the Intel® RealSense™ SDK

Introduction

Are you thinking about creating a simple application with RGB streaming that uses an Intel® RealSense™ camera and the Intel® RealSense™ SDK, or simply using RGB streaming in one of your applications? Do you want an easy-to-follow and easy-to-understand application that is direct and to the point without a lot of extra code that clouds up what you are trying to learn? Then you’re in luck, because that’s exactly what I’ve tried to do here: create a simple, yet effective sample application and document that describes how to use the Intel RealSense camera and SDK.

This sample was written using Intel RealSense SDK R4 Visual Studio* C# and tested with R5. It requires an Intel RealSense camera F200.

Project structure

In this sample application, I have tried to separate out the Intel RealSense SDK functionality from the Windows* Form GUI layer code to make it easier for a developer to focus on the SDK’s streaming functionality. I’ve done this by creating a C# wrapper class (RSStreaming) around some of the Intel RealSense SDK classes.

The Windows Form app contains only a few buttons and a PictureBox control to display the RGB stream.

Note that I’m not trying to make a bulletproof application. I have added some degree of exception handling; however, it’s up to you to ensure that proper engineering practices are in place to ensure a stable user-friendly application.

This project structure also relies on using Events to pass data around, which thus eliminates the need for tight coupling. A helper event class was created: RSNewImageArg which inherits from EventArg. It’s used to post the current frame from the camera back to the client form application.

Getting Started

To get started, you’ll need to have an Intel RealSense camera F200. You also need to have the Intel RealSense SDK version R4 or higher, and the appropriate Depth Camera Manager (DCM) installed on your computer. The SDK and F200 DCM can be downloaded here.

Requirements

Hardware requirements:

4th generation Intel® Core™ processors based on the Intel® microarchitecture code-name Haswell
8 GB free hard disk space
Intel RealSense camera F200 (required to connect to a USB 3 port)

Software requirements:

Microsoft Windows 8.1/Win10 OS 64-bit
Microsoft Visual Studio 2010–2015 with the latest service pack
Microsoft .NET* 4.0 Framework for C# development
Unity* 5.x or higher for Unity game development

RSNewImageArg.CS

RSNewImageArg derives from the C# EventArgs class. As you can see it’s a small wrapper that has one private data member added to it. The private Bitmap _bitMap holds the current bitmap that was extracted from the camera stream.

This class is used as an event argument when the RSStreaming class dispatches an event back to the Form class indicating that a new bitmap image is ready to display.

RSStreaming.CS

RSStreaming is a wrapper class, an engine so to speak around streaming RGB data from the Intel RealSense camera. I wrote the class with the following intentions:

Cleanly and clearly isolate as much of the Intel RealSense SDK functionality as possible away from the client application.
Try to provide comments in the code to help the reader understand what the code is doing.

The following describes each function that comprises the RSSpeechEngine class.

public event EventHandler<RSNewImageArg> OnStreamingImage;

The OnStreamingImage Event is used to trigger a message back to the client application letting it know that a new RGB bitmap image is ready to display. The client creates an event handler to handle the RSNewImageArg object.

public bool Initialized

Getter property used as a flag to indicate that the RSStreaming class has been initialized.

public bool IsStreaming

Getter property used as a flag to indicate that the RSStreaming class is currently streaming RGB data.

public void StartStreaming()

Checks to see if the class has been initialized and if not calls the InitCamera to ensure the class is up and running properly. Once this has been done, the function calls the _senseManager.StreamFrames( … ) function.

If you have done much reading about developing Intel RealSense applications, you have probably noticed that pulling data from the camera is often done in a while loop. For example, something like the following:

while (!Stop)
{
   /* Wait until a frame is ready: Synchronized or Asynchronous */
   if (sm.AcquireFrame(Synced).IsError())
      break;

  /* Display images */
   PXCMCapture.Sample sample = sm.QuerySample();

   /* Render streams */
   EventHandler<RenderFrameEventArgs> render = RenderFrame;
   PXCMImage image = null;
   if (MainPanel != PXCMCapture.StreamType.STREAM_TYPE_ANY && render != null)
   {
      image = sample[MainPanel];
      render(this, new RenderFrameEventArgs(0, image));
   }

   if (PIPPanel != PXCMCapture.StreamType.STREAM_TYPE_ANY && render != null)
      render(this, new RenderFrameEventArgs(1, sample[PIPPanel]));

   /* Optional: Set Mirror State */
   mirror = Mirror ? PXCMCapture.Device.MirrorMode.MIRROR_MODE_HORIZONTAL :
                                                                                                                                        PXCMCapture.Device.MirrorMode.MIRROR_MODE_DISABLED;
   if (mirror != sm.captureManager.device.QueryMirrorMode())
      sm.captureManager.device.SetMirrorMode(mirror);

   sm.ReleaseFrame();

   /* Optional: Show performance tick */
   if (image!=null)
      timer.Tick(PXCMImage.PixelFormatToString(image.info.format)+""+image.info.width+"x"+image.info.height);
}

This is a LOT of code to wade through. Now granted, they may be doing more than what my sample application does, but my point is that my application does not run a while loop like this. My application uses the StreamFrames(…) function. This function handles the while loop internally and for every frame triggers an event RSStreamingRGB will subscribe to. Essentially it works like this:

Kick off the stream PXCMSenseManager.StreamFrames(…).
Trap the event in an event handler.
When you’re done streaming, call the PXCMSenseManager.Close( ).

I like this approach, because I don’t want to have to manually deal with a while loop, knowing when and how to stop the loop. I would rather rely on the SDK to take care of that for me. When I talk about the InitCamera() function, you will see how this methodology is configured so I won’t talk about it here. Just make sure that you see how we can stream data and allow the SDK to handle the looping over the raw data coming from the camera.

Once the StreamFrames has been called, I then set the boolean flag _isStreaming to true allowing the class and client app to know that streaming has been started.

public void StopStreaming ( )

Stop streaming does the opposite of StartStreaming. It instructs the SDK to stop streaming data from the camera and calls Dispose() to destroy the objects the data.

private void InitCamera ( )

InitCamera() creates the PXCMSenseManager instance and enables the type of stream we want. As you can see I’m specifying a 320x240 color stream at 30 fps.

Recall what I said about being able to use an event from the PXCMSenseManger letting the class know when a new frame of RGB data is available. This is done using the PXCMSenseMananger.Handler event class. It’s a simple process: create an instance of the Handler class, assign it an event handler via the onNewSample, then initialize the PXCMSenseManager object; _senseMananger with the handler class.

Once this is completed, I set the _initialized flag to true. As previously mentioned, this flag is used to let either this class internally, or client app know that RSStreaming has been initialized.

private pxcmStatus OnNewSample( )

This is the event handler for the PXCMSenseMananger.Handler() object. Recall in the InitCamera() function I set the handler objects event handler to this function.

The event handler must adhere to a given function signature. The function must return a pxcmStatus value and takes two parameters:

Mid. The stream identifier. If multiple streams are requested through the EnableVideoStreams function, this is PXCMCapture.CUID+0, or PXCMCapture.CUID+1 etc.
Sample. The available image sample.

We need to convert the PXCMCapture.Sample object into a usable bitmap that the client application can use to display.

First I check to ensure that the sample.color object is not null and that the classes internal bitmap _colorImageData is not null as well. We need to ensure that our internal _colorImageData is not holding any data and to release it if it is.

Next we need to use the sample.colors object to populate the _colorImagedata. This basically is a metadata object about the PXCMCapture.Sample color object. Once we have that, we can tell it to create a bitmap for us specifying a size.

Once we have the bitmap and we know it’s not null, I trigger the OnStreamingImage event specifying the source of the event and a new RSNewImageArg object.

Finally, we MUST release the current frame from the PXCMSenseMananger object and as required by the function signature, return a pxcmStatus. I could have done some exception handling here, but I chose not to in order to keep things as simple as possible. If I had done some, I could have trapped it and chosen a different pxcmStatus to return, however, I’m just returning success.

private void Dispose ( )

Dispose() cleans up. I check to ensure that the manager is not null, that it was initialized and, if so, I call it’s dispose method. I check to ensure that RSStreaming’s bitmap is not null and dispose of it. Then I set everything to null.

MainForm.CS

The main form is the GUI that displays the RGB stream and allows you to control the RSStreaming object. It has two global variables: an instance of RSStreamingRGB and a bitmap. The bitmap will contain the current image from the current frame that’s sent by the RSStreamingRGB class.

public MainForm( )

The forms constructor. Creates a new RSSTreamingRGB object and gives the OnStreamingImage event an event handler.

private void btnStream_Click( )

The event handler when the Start Streaming button is clicked. Instructs the _rsStreaming object to start streaming by calling its StartStreaming() function.

private void btnStopStream_Click( )

The event handler when the Stop Streaming button is clicked. Instructs the _rsStreaming object to stop streaming by calling its StopStreaming() function.

private void UpdateColorImageBox( object source, RSNewImageArg e )

UpdateColorImageBox is the event handler for the _rsStream.OnStreamingImage event. It ensures that the newImage argument is not null, and, if not, assigns _currentBitMap to a new bitmap using the newImage as the source bitmap.

If I don’t create a new bitmap, the form’s _currentBitMap will be pointing back to the original bitmap that the SDK created. This can be problematic when calling the RSStreaming.Dispose method. The client has a picture box, the picture box has an image, and that image is coming from the SDK. When the form and picture box are still active, if I try to call RSStreaming.Dispose which releases SDK resources, I would get a crash because the picture box’s source image was now being disposed of.

After _currentBitMap has been assigned a new image, I call pictureBox.Invalidate() which forces the picture box’s Paint event to be triggered.

private void pictureBox_Paint( object sender, PaintEventArgs e )

This is the picture box’s paint event handler, which is triggered by the call to pictureBox.Invalidate(). It forces the picture box to redraw itself with the current source image.

First I check to ensure that the _currentBitMap is not null, and if not I set it to the most recent bitmap which is stored in _currentBitMap.

private void btnExit_Click( )

Easy enough. Simply calls Close(). No need to handle any clean up here because I ensure that this is happening in the MainForm_FormClosing method.

private void bMainForm_FormClosing( )

This is the forms event closing event handler. When the Close() method is called in any given function, the FormClosing event is called. I didn’t want to duplicate code so I simply put all clean up code here. I check to ensure that _rsStream is not null and that it’s streaming. If these conditions are met, I call _rsStream.StopStreaming(). There is no need to call a dispose method on _rsStreaming because it’s taken care of inside StopStreaming.

Conclusion

I hope this article and sample code has helped you gain a better understanding of how to use the Intel RealSense SDK to create a simple RGB streaming application. My intent was to show how this can be done in an easy-to-understand simple application that covers everything to be successful in implementing your own RGB streaming application.

If you think that I left out any explanation or wasn’t clear in a particular area, OR if you think I could have accomplished something in a better way, please shoot me an email at rick.blacker@intel.com or make a comment below.

About the Author

Rick Blacker is a seasoned software engineer who spent many of his years authoring solutions for database driven applications. Rick has recently moved to the Intel RealSense technology team and helps users understand the technology.

Contents