To: livid-gatos@xxxxxxxxxxxxxx, video4linux-list@xxxxxxxxxx,
linux-kernel@xxxxxxxxxxxxxxx
Subject: [RFC] alternative kernel multimedia API
From: volodya@xxxxxxxxxxxxxx
Date: Thu, 25 Oct 2001 13:06:12 -0400 (EDT)
Delivered-to: video4linux-list@xxxxxxxxxxxxxxxxxx
After looking at and working with Xv, v4l and v4l2 I became somewhat
dissatisfied with the current state of affairs. I have attached a
description of the API that would make (at least) me much happier.
I would very much appreciate comments from interested people..
thanks !
Vladimir Dergachev
Kernel multimedia architecture
** the latest version can be obtained from **
http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/gatos/km/km.rfc.txt
0) Motivation
v4l, v4l2 and Xv are all suffering from the same problem: attempt to fit
existing multimedia devices into a fixed scheme. The use of pre-defined
structs to describe parameters of device is inherently wrong because
these parameters vary widely. This leads to either bloating of the control
structures with parameters used only by few devices, proliferation of
device-specific ioctl and/or struct versioning. This also makes it increasingly
hard to implement support for new parameters.
The solution, IMO, is to move away from hard-coded models of multimedia
devices and instead allow greater flexibility to driver developers by
providing _symbolic_ interface.
The specific details follow, but the core idea is as follows:
a) for each hardware device/subsystem there is a "control" device
which user application can use to set/query parameters. For example,
to set hue it can issue a command like:
HUE=7\n
to be replied with
+\n
(indicating that the command succeeded)
or
-HUE=INVALID
if HUE attribute is not available.
To query picture size it can issue:
?WIDTH,HEIGHT\n
To be replied with
+WIDTH=640,HEIGHT=480\n
In case the device supports it.
b) for some devices (all capture or output devices) there is one (or more ?)
"data" device.
The data device is meant for high-speed transfer of data. The data is
accessed by memory mapping the data device (the size is found out via
control device).
The beginning of the memory mapping is a control/status struct whose
primary use is exchange of information and synchronization between
application and kernel driver.
For example, for a frame grabber the control struct will have (at least)
three fields for each buffer: timestamp, number, and busy.
"busy" will be set when the buffer is acquiring new data,
"timestamp" will be an integer value monotonically increasing with time
and "number" will be the serial number of the captured frame.
When capturing video the application will read data from the buffers
until it encounters one with smaller serial number and/or marked as busy.
It then has an option to do something else and/or issue a request
on control interface to notify when more data is available.
The advantage of this scheme is elimination of context switches
due to ioctl and selects, but, more importantly, elimination of the
numereous ioctls whose sole purpose is to transfer data between
application and the kernel.
c) So far we replaced numerous ioctl with a control interface and a control struct.
However the question of the format of the control struct still remains.
The answer is that the format itself is quieried via control interface:
?GLOBAL_PARAM_OFFSET returns the offset of the global parameter area
?GLOBAL_PARAM_FORMAT returns the interface class of the global
parameter area followed by the lists of fields
and offsets within that area
?BUFFERS_COUNT returns the number of buffers
?BUFFER_OFFSET,BUFFER=X returns the offset of the buffer
?BUFFER_PARAM_OFFSET,BUFFER=X returns the offset of the buffer-specific
parameter area
?BUFFER_PARAM_FORMAT,BUFFER=X returns the interface class of the
buffer specific parameter area
followed by list of fields and offsets
within that parameter area
d) Questions that occured to me while I was writing this:
Q1) Should something that complex be in the kernel?
A1) The control language is very simple. An example of
something similar would be software modems and ISDN
which must be able to parse AT command set.
Q2) Is it really necessary to bloat video drivers with
parsing functions
A2) the idea is that there will be a library of functions
used by all drivers simultaneously. The driver code
will only need to declare it structures and let the
library know about them - the actual interface code
will be common to all drivers
1) User-space interface basics
/devfs/multimedia/devices - a text file that enumerates known devices
/devfs/multimedia/control/deviceX - control devices
/devfs/multimedia/data/dataY - high-speed data devices
(X and Y are numbers)
Protocol:
a) user application reads /etc/fstab (/etc/mtab) to find out where /devfs
is mounted.
b) user application reads /devfs/multimedia/devices to find out about
devices present
c) to open a device it opens /devfs/multimeda/control/* and negotiates
parameters/performs control functions
d) when data transfer is needed it queries /devfs/multimedia/control/deviceX
about port number for the data interface and opens
/devfs/multimedia/data/portY for R/W or R depending on intended purpose
It then queries /devfs/multimedia/control/deviceX about the size of
data interface region of portY and memory maps the area with R/W or R
privileges.
Next it queries control deviceX about the size and format of control
section of data interface region of portY which is always located in
the beginning of data interface region.
It obtains buffer information from the control section of data interface
region.
e) to make actual transfer it uses interface specific protocol
2) User-space interface: control interface protocol
control interfaces and /devfs/multimedia/devices are character based devices.
The negotiation, command issue and query is conducted in the following fashion:
Each request/query takes one line and is terminated by '\n'. The request
consists of status character and several fields. Examples:
a)
?PCI_ID\n
b)
REPORT_VSYNC=1\n
(space)
c)
?FRAME_RATE,WIDTH=640,HEIGHT=480\n
d)
+\n
e)
-PCI_ID=INVALID\n
f)
:VSYNC=4\n
Status character Description
------------------------------------------------------------------------------
? Perform query, no device state is changed
' ' (space) Execute. Parameters are set and/or action is
performed.
+ Reply: query/command succeeded. Addition information
may follow (esp. in the case of query)
- Reply: query/command failed. In particular this
is returned if certain fields are not recognized
by the driver. In this case the fields are always
appended as in e) (for example this could be
returned for a USB device).
: Indicates out-of-sequence reply. Normally each
query/command is responded to with a + or -
reply. However, sometimes that driver needs to
report a state change (i.e. in example f it
reports that 4 interrupts occured since
last report) without propmting of user-space
application.
/devfs/multimedia/devices format
Each line represents one device or subsystem of a device. The following fields
are required:
DEVICE_ID an identifier specific to this device
it should be the same if the device is
plugged into a different port or a different
computer
LOCATION_ID an identifier specific to the location of the
device. All devices plugged into the same
port should have the same LOCATION_ID.
Example: PCI_ID for PCI cards
INSTANCE_ID an identifier specific to the device
it is required to be unique for each device
in the system
All *_ID fields should be composed by attaching different strings with "-"
so that the user can use pattern matching ala X fonts.
The following fields are optional:
CONTROL specifies the number of the control
interface used for this device
SUBSYSTEM_OF specifies INSTANCE_ID of the device this one
is a subsystem of. Useful for devices that
provide more then one hardware interface.
To: video4linux-list@xxxxxxxxxx
Subject: Re: [RFC] alternative kernel multimedia API
From: Justin Schoeman < justin@xxxxxxxxxxxxxxxxxxxx>
Date: Fri, 26 Oct 2001 08:54:14 +0200
Delivered-to: video4linux-list@xxxxxxxxxxxxxxxxxx
Organization: University of Pretoria
User-agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:0.9.2) Gecko/20010704
volodya@xxxxxxxxxxxxxx wrote:
> After looking at and working with Xv, v4l and v4l2 I became somewhat
> dissatisfied with the current state of affairs. I have attached a
> description of the API that would make (at least) me much happier.
> I would very much appreciate comments from interested people..
> thanks !
> Vladimir Dergachev
> ------------------------------------------------------------------------
> Kernel multimedia architecture
> ** the latest version can be obtained from **
> http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/gatos/km/km.rfc.txt
> 0) Motivation
> v4l, v4l2 and Xv are all suffering from the same problem: attempt to fit
> existing multimedia devices into a fixed scheme. The use of pre-defined
> structs to describe parameters of device is inherently wrong because
> these parameters vary widely. This leads to either bloating of the control
> structures with parameters used only by few devices, proliferation of
> device-specific ioctl and/or struct versioning. This also makes it increasingly
> hard to implement support for new parameters.
> The solution, IMO, is to move away from hard-coded models of multimedia
> devices and instead allow greater flexibility to driver developers by
> providing _symbolic_ interface.
...
I don't know - I don't see anything in the interface design that can do anything more
than v4l2. In fact, for high speed capturing I can see a number of possible catches.
I would like to see a detailed explanation of how the capabilities differ from those
of v4l2.
(I also doubt that an interface this complex would ever make it into the kernel.)
-justin
To: video4linux-list@xxxxxxxxxx
Subject: Re: [RFC] alternative kernel multimedia API
From: Gerd Knorr <kraxel@xxxxxxxxxxx>
Date: 26 Oct 2001 10:17:05 GMT
Delivered-to: video4linux-list@xxxxxxxxxxxxxxxxxx
Newsgroups: lists.linux.video4linux
Organization: SuSE Labs, Außenstelle Berlin
User-agent: slrn/0.9.7.1 (Linux)
> I don't know - I don't see anything in the interface design that can do
> anything more than v4l2. In fact, for high speed capturing I can see a
> number of possible catches.
I agree. The contingous capture basically removes all syncronisation
between driver and application and thus leads to plenty of race
conditions and other headage. Applications can't stall capture if they
can't keep up, instead old (maybe not processed yet) data gets
overwritten.
And if you want something generic to control devices, the v4l2 controls
allow you to do that.
Also note that devfs is still experimental, I don't like the idea to
build something on top of devfs. /me stopped using devfs recently btw.
Gerd
--
Netscape is unable to locate the server localhost:8000.
Please check the server name and try again.
To: video4linux-list@xxxxxxxxxx
Subject: Re: [RFC] alternative kernel multimedia API
From: volodya@xxxxxxxxxxxxxx
Date: Fri, 26 Oct 2001 07:56:16 -0400 (EDT)
Delivered-to: video4linux-list@xxxxxxxxxxxxxxxxxx
In-reply-to: <3BD90896.6090909@xxxxxxxxxxxxxxxxxxxx>
On Fri, 26 Oct 2001, Justin Schoeman wrote:
> volodya@xxxxxxxxxxxxxx wrote:
>
> > After looking at and working with Xv, v4l and v4l2 I became somewhat
> > dissatisfied with the current state of affairs. I have attached a
> > description of the API that would make (at least) me much happier.
> >
> > I would very much appreciate comments from interested people..
> >
> > thanks !
> >
> > Vladimir Dergachev
> >
> >
> >
> >
> >
> > ------------------------------------------------------------------------
> >
> > Kernel multimedia architecture
> >
> > ** the latest version can be obtained from **
> >
> > http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/gatos/km/km.rfc.txt
> >
> > 0) Motivation
> > v4l, v4l2 and Xv are all suffering from the same problem: attempt to fit
> > existing multimedia devices into a fixed scheme. The use of pre-defined
> > structs to describe parameters of device is inherently wrong because
> > these parameters vary widely. This leads to either bloating of the control
> > structures with parameters used only by few devices, proliferation of
> > device-specific ioctl and/or struct versioning. This also makes it increasingly
> > hard to implement support for new parameters.
> >
> > The solution, IMO, is to move away from hard-coded models of multimedia
> > devices and instead allow greater flexibility to driver developers by
> > providing _symbolic_ interface.
>
> ...
>
> I don't know - I don't see anything in the interface design that can do
> anything more than v4l2. In fact, for high speed capturing I can see a
> number of possible catches.
>
> I would like to see a detailed explanation of how the capabilities
> differ from those of v4l2.
The major difference is in increased flexibility and stability of
kernel-user space interface.
On the application side:
* ability to utilize advanced driver features without explicit
knowledge of their behaviour
* wider compatibility without the need for recompile
On the kernel side:
* elimination of interfaced specific headers
On the driver side:
* be compatible with a wide range of applications
* introduce support for new features without the need to modify
kernel interfaces
The last point is the very important in my opinion. For example, in
current specification, v4l2 has no support for TV-out in graphics cards.
It has no support for setting complex parameters like gamma tables.
Using memory-mapped buffers requires device specific ioctls.
The goal is to create an interface that does not rely on structures
defined in kernel headers for communication.
>
> (I also doubt that an interface this complex would ever make it into the
> kernel.)
It is not very complex. It is much simpler than software modem driver,
for example.
Ultimately these questions are best answered with a sample implementation.
Would you have any other concerns/suggestions ?
Vladimir Dergachev
>
> -justin
>
>
>
> _______________________________________________
> Video4linux-list mailing list
> Video4linux-list@xxxxxxxxxx
> https://listman.redhat.com/mailman/listinfo/video4linux-list
>
To: video4linux-list@xxxxxxxxxx
Subject: Re: [RFC] alternative kernel multimedia API
From: Alan Cox <alan@xxxxxxxxxx>
Date: Fri, 26 Oct 2001 08:22:27 -0400 (EDT)
Delivered-to: video4linux-list@xxxxxxxxxxxxxxxxxx
In-reply-to: <3BD90896.6090909@xxxxxxxxxxxxxxxxxxxx>
from "Justin Schoeman" at Oct 26, 2001 08:54:14 AM
I do it when I get time - it really is erratic updates not a caching bug 8)