My idea was to do something similar to the framebuffer device, using map to access image data and ioctl's to control camera parameters and wait for a frame. Is this mostly how v4l works? Do you see a need for something more? Not sure what other aspects of video handling require a specific solution.
Best, Matias