Video for Linux API Proposal

These proposals do not cover the entire Video for Linux spec. The rest of the spec I consider OK as is, or I have no opinion on it.

Multiple Devices per System

Drivers should be able to support multiple devices, as long as the hardware can do it. It's trivial if the driver writer keeps all global variables in a device structure that begins with a struct video_device. All entry points into the driver pass in a pointer to this structure.

Multiple Opens per Device

Supporting multiple simultaneous capture operations on the same device is not practical because there is no open handle which can be used to differentiate the different capture contexts, and because streaming or frame buffer capture is impractical for more than one instance.

However, it would be really good to support two opens on a device for the purpose of having one open be for capturing, and the other for a GUI control panel application that can change brightness, select inputs, etc. along side the capturing application. A standard video control panel that works with all Video for Linux devices and that can run concurrently with any capturing application would be very cool, and would relieve all the application developers from each having to incorporate their own control panel.

Query Capabilities - VIDIOC_G_CAP

This ioctl call is used to obtain the capability information for a video device. The driver will fill in a struct video_capability object.

Note that the minimum and maximum image capture dimensions are for comparison purposes only. The actual maximum size you can capture may depend on the capture parameters, including the pixel format, compression (if any), the video standard (PAL is higher resolution than NTSC), and possibly other parameters. Same applies to maximum frame rate. The minimum and maximum sizes do not imply that all combinations of height/width within the range are possible. For example, the Quickcam has three settings.

Capture to a frame buffer might not work depending on the capabilities of the graphics card, the graphics mode, the X Windows server, etc.

Capture Image Format - VIDIOC_G_FMT, VIDIOC_S_FMT

The capture format defines how the image is layed out in the capture buffer. It defines the dimensions of the image and the pixel format or compression format. The information is exchanged in a struct video_format object.

Devices will not be able to support every combination of width and height. The driver will find the width and height compatible with the hardware which are as close as possible to the requested width and height without going over in either dimension. Applications must do a VIDIOC_G_FMT to get the actual dimensions granted and make sure they are suitable.

The depth is the amount of space in the buffer per pixel, in bits. The pixel information may not fill all bits allocated, e.g. RGB555 and RGB32. Leftover bits are undefined. For planar YUV formats the depth is the average number of bits per pixel. For example, YUV420 is eight bits per component, but the U and V planes are 1/4 the size of the Y plane so the average bits per pixel is 12. The pixelformat values and flags values are defined in the tables below.

Bytesperline is the number of bytes of memory between two adjacent lines. Since most of the time it's not needed, bytesperline only applies if the \ FMT_FLAG_BYTESPERLINE flag is set. Otherwise the field is undefined \ and must be ignored. For YUV planar formats, it's the stride of the Y plane.

Sizeimage is usually \ width*height*depth for \ uncompressed images, but it's different if bytesperline is used \ since there could be some padding between lines. Sizeimage is \ ignored on set. A capture operation such as read() is allowed to fail if the \ buffer is smaller than sizeimage since a partial image read may be \ nonsensical or impractical to implement.

An interlaced image will have "comb" or "feathering" artifacts around moving objects. If the \ FMT_FLAG_INTERLACED flag is not set on \ VIDIOC_S_FMT, then the driver is not permitted to capture interlaced \ images. If the flag is set then the driver may (but is not required to) \ capture interlaced images if the requested vertical resolution is too high for a single field. FMT_FLAG_INTERLACED is set on return from VIDIOC_G_FMT only if the driver is actually going to capture interlaced images.

Compressed Capture - VIDIOC_G_COMP, VIDIOC_S_COMP

These ioctls set additional capture parameters needed for compressed capture. They both pass the information in a struct video_compression object. The keyframerate field only applies to temporal compression algorithms. The quality factor ranges from 0 to 65535.

Reading Captured Images - read()

This capture mode is supported if the VID_TYPE_CAPTURE flag is set in the struct video_capabilities. Each call to read() will fill the buffer with a new frame. The driver may fail the read() if the length parameter is less than the required buffer size specified by the VIDIOC_G_FMT ioctl(). This is reasonable since each call to read() starts over with a new frame, and a partial frame may be nonsense (e.g. for a compressed image) or impractical or inefficient to implement in the driver.

Non-blocking read() mode is supported in the usual way. Read() does not work if either streaming capture or hardware frame buffer capture is active.

Capturing to a Hardware Frame Buffer - VIDIOC_G_FBUF, VIDIOC_S_FBUF, VIDIOC_G_WIN, VIDIOC_S_WIN, VIDIOC_CAPTURE

This capture mode is supported if the VID_TYPE_FRAMEBUF flag is set in the struct video_capabilities. [This is very much like the current spec. We might add some get- capture- card- capabilities thing. For example the card I have can only DMA YUV4:2:2 data.]

VIDIOC_S_FBUF sets the frame buffer parameters. VIDIOC_G_FBUF returns the current parameters. The structure used by these ioctls is a struct video_buffer. Ideally the frame buffer would be a YUV 4:2:2 buffer the exact size (or possibly with some line padding) of the capture. It could also be the primary graphics surface, though. You must also use VIDIOC_S_WIN to set up the placement of the video window in the frame buffer.

Note that the buffer is often larger than the visible area, and so the fmt.bytesperline field is most likely valid. XFree86 DGA can provide the parameters required to set up this ioctl.

VIDIOC_G_WIN and VIDIOC_S_WIN work just like the existing VIDIOCGWIN and VIDIOCSWIN ioctls. Except:

1. The width and height fields of the struct video_window reflect the width and height of the image on the screen, not the width and height of the capture. In other words the captured image may appear stretched on screen.
2. These ioctls only apply to frame buffer capture. The capture dimensions are set with the VIDIOC_x_FMT ioctls.

VIDIOC_CAPTURE is the same as the existing \ VIDIOCCAPTURE ioctl.

Capturing Continuously to Pre-Allocated Buffers - VIDIOC_G_STREAM, VIDIOC_S_STREAM, VIDIOC_MCAPTURE, VIDIOC_SYNC

This capture mode is supported if the VID_TYPE_STREAM flag is set in the struct capture_capabilities.

I need help on this one! I guess similar to the way the Bt848 driver is doing it now is the best we can do. The driver allocates page-locked buffers in kernel space. The app uses VIDIOC_S_STREAM to request how many buffers it wants, and VIDIOC_G_STREAM to find out how many buffers the driver agreed to allocate. The layout of the buffers is defined with VIDIOC_S_FMT. If the format changes, the buffers are freed. Mmap is used to map the buffers to user space. Calls to VIDIOC_MCAPTURE add buffers to a queue. When a buffer is filled the app is notified (VIDIOC_SYNC or select() unblocks?). Somehow the caller needs to know which buffer is ready. When the app has consumed the data it requeues the buffer again.

When a buffer is filled, the caller will need a way to get the number of bytes in the buffer (varies for compressed formats) and the time stamp of when the frame was captured. I guess the caller can do a VIDIOC_G_STREAM after a frame completes to get this information. Ok, how about VIDIOC_G_STREAM returns a structure something like this:

Waiting for Frames Using select()

The driver supports the select() call on its file descriptors if the VID_TYPE_SELECT flag is set in the struct capture_capabilities. Details to come... (Thanks, Aaron)

Capture Parms - VIDIOC_G_PARM, VIDIOC_S_PARM

This is to control various parameters related to video capture. These ioctls use struct video_parm objects. The microsecperframe field only applies to read() and streaming capture. Capture to frame buffer always runs at the natural frame rate of the video.

It is worth stressing that switching the video standard is a big deal. Many capabilities of the capture card depend on the video standard selected, including the image resolution and frame rate. After changing the standard the capture dimensions, required image buffer size, or other capture parameters may have changed. The caller should re-set-up the capture. [The driver may only allow a standard change when there is only one open on the device.]

[Logically, the standard should be on a per-input basis, but since changing the standard is so dangerous, and we want to be able to have a separate control panel that can select inputs independent of a capturing application, we don't want an input change to possibly change the standard. Also mixed standard devices on one system is extremely rare. So I put it here.]

Flags for the capturemode and capability fields [not done yet]

Settings for the standard field [not done yet]

Video Inputs - VIDIOC_G_INPUT

This ioctl retreives the properties of a video input into a struct video_input object. Before calling VIDIOC_G_INPUT the caller fills in the number field to indicate which input is being queried.

Video Tone Controls - VIDIOC_G_PICT, VIDIOC_S_PICT

These get or set the video tone control settings for the currently selected input. The settings are passed in a struct video_picture object. There are separate tone control settings for each input, so an application must do a VIDIOC_G_PICT after changing the input. All values are scaled between 0 and 65535. 32768 is always a safe neutral position.

Tuning - VIDIOC_G_TUNER, VIDIOC_S_TUNER

Let's fill this in. Something like the existing VIDIOxTUNER is probably pretty close. Someone wanted to add fine tuning hint feedback from the tuner. We can probably get rid of the FREQ ioctls if we add a frequency field to struct video_tuner.

Compression/Decompression and Effects

This refers to performing operations on video frames that have been captured previously. This does not refer to compressed capture. It's possible that a device implementing some of these functions may not have capture capability at all.

Ideas?