Please note that this project is no longer maintained by Sensoray — it is provided here in the hope it will be useful to developers.

Stream Multiplexing

A common application for capture boards is to save paired audio/video streams to disk. Saving the video stream by itself is straightforward because it can be stored directly to disk as a Video Elementary Stream, or VES. In the case of an a/v stream pair, however, it is desirable to save the audio and video streams together in a single file. This is accomplished by first multiplexing the streams together, with appropriate timestamps, and then saving the resulting multiplexed stream to disk.

Multiplexing is a complex and difficult programming endeavor, so it's usually impractical to develop a multiplexer from scratch. There are many commercial multiplexers available for DirectShow, but most are closed source and thus present formidable obstacles to the beleaguered developer who is trying to debug problems involving a live, multiplexed stream. Furthermore, the standard DirectShow multiplexer filters do not support many of the popular compression formats such as MPEG-1 and MPEG-2. Clearly, it is advantageous to use a versatile, open-source multiplexer if one is available.

Enter VLC

The VLC media player — also known simply as VLC — is an open-source software project that is licensed under the terms of the GNU General Public License and developed by the VideoLan team (www.videolan.org). VLC implements a rich set of stream processing functions for encoding and decoding media streams, with support for a wide variety of compressed stream formats including MPEG-1, MPEG-2 and MPEG-4 (MP4S or Microsoft MPEG-4), as well as uncompressed audio.

Most importantly, VLC has built-in support for stream multiplexing, with the ability to generate a variety of multiplexed stream formats including MPEG system stream, MPEG program stream and MPEG transport stream. A multiplexed stream produced by VLC can easily be saved to a file or streamed onto a network.

The VLC-to-DirectShow Connection

Because VLC is open source and implements a high quality, flexible and reliable multiplexer, Sensoray chose it as the foundation for a DirectShow multiplexer filter. VLC, however, is an application program, which is to say that it is not a DirectShow filter.

VLC is designed to work in conjunction with plug-ins and modules. For example, video streams enter VLC through an access module. Ironically, VLC has a DirectShow access module, but this module can't be used to insert VLC into a filter graph as a separate filter. Also, VLC provides access to DirectShow drivers, but such drivers typically support only uncompressed capture cards.

In order to work around these issues and make it possible for VLC's multiplexer to be used with DirectShow, Sensoray developed the Sensoray VLC Bridge, a DirectShow filter that incorporates the VLC libraries. This filter serves as a bridge from DirectShow to VLC, thus exposing VLC's multiplexer functionality in the form of a DirectShow filter. It is a very useful standalone filter and, in the spirit of the open source movement (and as required by GPL), Sensoray provides all source code for the filter on its website.

Installing and Building the VLC Bridge

This is only a brief summary for developers; please read the details in the source code.

The DirectShow filter is built in the SensorayVlcBridge subdirectory through Microsoft's Visual Studio tool. Libvlc is built using MinGW, the minimalist GNU for Windows(www.mingw.org). Compiling libvlc in MinGW is beyond the scope of this application note (many resources for this are available on the web). The “configure” script is in the file configure_vlc.bat. Unnecessary options such as active-x (for display) and MPEG encoding/decoding were disabled. Another README exists in the root directory for build problems. Installing the bridge filter is done with “regsvr32 srayvlcbridge.ax”. The VLC library must be in the same directory when the filter is registered.

In order to co-exist with other versions of VLC, the plugins are installed in a common directory off of the Windows directory. This directory is called “dshow_vlc_plugins” and must exist with the plugin dlls before the VLC Bridge filter is used.

On the user level, the bridge is used in a DirectShow FilterGraph. A simple example using the GraphEdit tool is shown below. Please note that GraphEdit should only be used for debugging purposes, not for finished products.

First, the Mpeg and Audio source filters are created.
This example shows Sensoray's 2250 USB capture device.

Second, the VLC Bridge is inserted.

Third, click Insert Filter. GraphEdit will prompt for a filename. Enter “mpeg2_test.mpg”, and click Open.

Finally, connect the pins. Normally, this is done using C or C++ code in a
DirectShow application using DirectShow interfaces such as IGraphBuilder.

The DirectShow filter graph is now finished and ready for use. The graph above shows the 2250 audio connected directly to the VLC bridge muxer filter. In this example, VLC does the audio encoding. An MPEG-2 program stream is saved to a file when the graph is started. Pressing the green arrow starts the graph running.

To stop recording and close the file, simply stop the graph. In GraphEdit, this is done with the stop button. In a DirectShow application this is done with the IMediaControl interface.

There is one caveat with the filter. Currently, the VLC filter requires disconnecting the pins, deleting the filter and re-adding it before starting a new file. This behavior is slightly non-standard, but is easily handled by a C/C++ application.

Where are the output pins for the VLC filter?

The Sensoray VLC Bridge filter is a sink filter. It is the end of the filter graph and this filter releases samples. It has the same interface as IfileWriter, the standard FileWriter filter. In the DirectShow world, it is a combination Multiplexer and FileWriter.

If we use this filter, do we need to release our code under the GPL?

GPL licensing is a complex topic. Sensoray does not provide legal advice about GPL or otherwise. It is your responsibility to make sure you respect the license agreement. The source code for this filter is freely available and the filter is useful as a standalone product. All interfaces to the filter are through DirectShow.

If you use any of the headers from this project, you inherit the GPL license and accordingly must release your source code into GPL. Additionally, if you use this filter, you must show the GPL license at install, mention the VideoLAN team and clearly state that VLC is not your copyright. JB Kempf, a lead developer for VLC has stated on his blog that you can build a proprietary application using libvlc that uses standard commucation (D-BUS or Active-X), but never using native GPL source (C/C++/C#).

Can I use the filter for streaming video over RTP/UDP? Can I do transport stream?

Yes. Since the filter uses VLC, you may change the output of the libvlc calls from file output to streaming output. Sensoray provides the source code “as is” under the GPL.

What formats does this filter support? Can I do MPEG4?

Currently, this filter supports MPEG1 and MPEG2 video streams with PCM or MPEG1Layer2 audio streams. It does not support audio elementary stream(AES) or audio by itself. This filter does not support MPEG4 as is. Sensoray uses the standard Microsoft AVI multiplexer for MPEG4 (MP4S) multiplexing.