Transcript
Investigating CIRO Tracking Camera Provisioning Will Baer As part of the CIRO project, we investigated ways of inexpensive provisioning of CIRO-style functionality to users of our environment. One aspect that received significant attention was provisioning of the tracking camera that would allow sensing of the gestures made. Prototypes of CIRO utilized re-purposed Sony PS3 gaming cameras. These cameras had specifications of 120 frames per second (“fps”) at QVGA (320 x 240 pixel) resolution and detected colour. We speculated that CIRO resolution and performance could be significantly improved by a) moving to a VGA (640 x 480) image resolution and b) removing the Bayer RGB (Red, Green, Blue) colour-filter pattern superimposed on the Sony camera’s image pixels and replacing it with a monochromatic “band-pass” optical filter that would remove all light except for the frequency of interest. Since our system worked with monochromatic light, the colour filters simply worsened the pixel signal-to-noise ratio. As well, the factor of four in image resolution improvement (2x in width, 2x in height) would be welcome as long as it could come without frame rate degradation. While cameras of such capability existed, we felt that their cost (at that time starting from about $900 per camera) could prove a barrier to adoption of the CIRO system. This spurred a research project to investigate the possibility of creating a sort of “open-source” reference design that (with some support) would allow CIRO users to build their own cameras using open-source plans and parts lists. Our target production price for these cameras (ex. labour) was $200. This work proceeded simultaneously in three directions over several months: a) a survey of available chips sets, commercial reference designs, and parts costs; b) additionally, we approached provisioning from a “can we hack existing, highvolume-production, video components to get what we need” point of view; and c) a high-level survey of trends in imaging technology development was conducted. The findings of these three parallel investigations, arising as they did from the entwining of our three threads of investigation, are discussed below along with a list of what seem particularly important recent/upcoming events: Survey of Available Imaging System Base Technologies Initially, it seemed like a very real possibility that such an “open-source hardware” camera could be created. Texas Instruments (“T.I.”) chipsets and reference designs were available claiming to provide the possibility of a “$150” imaging solution. However, upon deeper examination, it became evident that this cost would really only apply to production runs of large numbers of cameras. Further, much of this work was done in the vein of security surveillance web-cams, with motion JPEG or MPEG image compression applied
at chip level before data transfer is done. While useful in many cases, it is not at all clear that such compression might not so compromise image quality for a movement tracking application as to render it useless. Still further, the use of higher frame rates requires higher pixel clocking frequencies. This in turn necessitated either more complex circuit boards (with better shielding and circuit traces specially shaped so as to minimize radiation of EM noise) or microminiature pre-assembled “System-on-Chip” (“SoC”) modules made at the factory so that they could simply be plopped down on a circuit board with much of their internal complexity masked by the tight die-to-die SoC interconnects and shielded packaging. Exemplary of the latter approach was an offering from OmniVision of their “Camera Cube” technology featuring VGA imaging sensors packaged into a lensed (63 degrees FOV) assembly with dimensions of only 2.5 x 2.9 x 2.5 mm. While everything else was correct from our perspective, the Camera-Cube technology was, sadly, limited in frame rate to 30 Hz. Interestingly – and germane to our high-level trend survey – the form-factor of the imaging sensor was 1/13”. This is important because semiconductor die size tracks very closely with part cost/price; the more dice that can be cut from a standard sized fab wafer, the cheaper the cost of fabrication. Recently, OmniVision announced a 1/6” sensor claimed to be capable of 120 fps at VGA resolution; it is scheduled for production starting in June of 2010. This sort of sensor, with its significantly smaller die size (and therefore presumably smaller price), is just the sort of innovation we have been waiting for. However, an additional observation that arose from our technology search was that, due to the miniaturization of integrated circuit chip packages, it is already close to impossible to mount and solder such chips without the use of expensive robotic “pick and place” machines. The immediate implication of this from an “open-source hardware” perspective is that circuit boards would have to come with chip parts already mounted and tested; it would not be possible to ship kits of parts and circuit boards to end-users and expect them to be able to assemble them. Thus manufacturing activities of some sort would have to be undertaken or contracted to provision these pre-mounted boards. As we looked into the technology of the imaging chips themselves, there were also subtleties that became apparent. Two types of electronic shuttering system are commonly used on such chips: a) a “rolling” shutter where rows of pixels are exposed one line at a time in an endless tail-chase of rows on the image sensor from top to bottom to top again, and b) a system where all pixels on the chip’s imaging array are exposed at the same time for a full frame’s worth of light. In the former system, the effect is to smear the exposure of the image through time since different rows of one image frame were exposed at different moments over the course of the image frames aggregate exposure time. While inconsequential for many applications, such an exposure method could cause problems for tracking systems and should be avoided. However, the downside to option b) above (the full frame exposure approach) is one of chip real estate and, ultimately, chip cost. In order to expose all pixels simultaneously, additional buffer amplifiers are required to hold the pixel charges while they are sequentially read-out after frame exposure is complete. These additional electronics take up space and result in larger chip die sizes and thus higher costs.
On yet a third hand, however, at higher frame rates, this time “smear” becomes less important, as movements or gestures are more static from frame to frame. Finally, our review of the available technology also pointed up that it was not just the technology of imaging systems per se that needed to be taken into consideration. In order to have a system like CIRO be available at low cost, one must also take into account the cost/availability of both data transfer protocols and of image-processing computing capability. For example, a commonly used method of high-speed image transfer is gigabit Ethernet. One gigabit of data transfer is approximately 80 megabytes per second at 12 bit pixel resolution (which would give a reasonable dynamic range of 72 dB); at this speed, a 4 GHz PC would have only about 50 CPU machine instruction cycles per pixel unless recourse was found in some sort of pre-processing or off-loaded (i.e. non-CPUbased) processing of image frames. For a 640 x 480 VGA imaging system, 1 gigabit would correspond to about 270 fps of uncompressed image transfer. This implies that any image processing at rates much faster than around our target of 120 fps must wait on faster transfer and computational technologies as well. Possibilities of Hacking Existing Imaging Systems In our evaluation, we looked at the possibility of being able to adapt existing imaging devices to our needs. Given the odd realities of electronic manufacturing that presently produce some extremely sophisticated consumer devices at low retail cost, the appeal of this approach was great. One initial candidate was the LG “Viewty” cell phone, which featured a 120 fps camera built into it. However, as we looked into the possibilities of doing this, it became clear that most (if not all) camera and phone hacks focus on allowing export of recorded video, not on timing firmware modifications. Unwilling to embark on an unpaid career as a mobile-computing firmware developer, we abandoned this approach. Subsequent to this, we also confirmed that the 120 fps frame rate was available only in QVGA (320x240) resolution, thus confirming our decision to abandon this avenue. Imaging System Trends Regarding imaging systems, our high-level gestalt of the industry revealed the following insights: Frame-rate limitations for VGA-resolution imaging systems arise not only from the imaging systems themselves, but also from the data transfer protocols available. For example, gigabit Ethernet is limited to about 270 fps at full VGA resolution. If one is concerned with tracking/predicting cost and functionality of high frame-rate tracking systems, the implication of this is that it is not just imaging system innovation that must be monitored but also that of the data transfer protocols, plus the ability of computer systems to process such data in real-time. • The pace of imaging system innovation is accelerating. This is being driven by commercial systems drivers such as: a) mobile computing (cell phones, but •
also iPad, iPhone, Blackberry-style devices, and tablet/laptop/palmtop computers); b) gaming (both from dedicated gaming consoles and home computers); c) social networking and online interactivity using home computers equipped with varying sorts of “web-cam” imaging systems, and d) network-based surveillance camera systems • Smaller imaging chip die sizes for given levels of performance are continuing to arrive at a record pace, implying a continuing downtrend in imaging system cost. • Imaging systems development is increasingly driven by (among others) gaming. The needs of gaming very much parallel CIRO’s needs, particularly the need for higher frame rates. It is expected that there will be gamingdriven availability of high-frame-rate cameras in the near (1-3 year) future. • Much like with the cell-phone business/pricing model, gaming-driven imaging is being provided at, or below, cost. This is to encourage user adoption of gaming platforms, thus locking in future software sale revenue streams. A side benefit is that those able to piggyback on such developments effectively ride “for free”, benefiting from the lower hardware cost without ultimately paying the price in software purchases. Recent and Upcoming Events The following events are expected to have a significant impact on CIRO-related imaging technologies: • Late Q1, 2010 announcement of full-HD (1080p) resolution imaging sensor of just 1/6” form factor by OmniVision to be in production by June, 2010 sets the tone for the continued pace of image chip miniaturization. It comes with the promise of full VGA resolution imaging at 120 fps. Given the chip die size, the price of these chips should also be good and should facilitate low-cost imaging systems built around them. •
Announcement of Sony PS3 “Move” gaming platform based on optical sensor tracking in Q1/Q2 2010 (available in fall 2010)
•
Mid-June, 2010 unveiling of Microsoft “Project Natal” gaming platform
Final Summary of Findings At the end of our investigation, we concluded that it was probably not wise to pursue the concept of “open-source” hardware in relation to high-speed video imaging technologies. The reasons for this were as follows: Because imaging system chip development is moving quickly, any opensource system that didn’t also keep up technically would likely be orphaned by its community in a few years and replaced by more functional commercial offerings. • Manufacturing and alignment of small-form-factor imaging systems is not possible at home, it too requires expensive, accurate, equipment and must be done in a factory setting. Thus such continued R&D development would require lab space and test equipment. •
Imaging systems development is already a fast moving field and the pace is accelerating, driven by new impetus coming from large consumer markets in mobile computing, gaming, et al. Any attempt to provide open-source hardware would have to be bulwarked by a large community of developers. Unlike say, the Linux development community, which has created a software platform that competes well with commercial offerings, creation of imaging systems demands costly chip placement tools, test equipment, and board fabrication access; the barriers to entry into this realm seem high as does the ongoing cost of continued development and improvement in order to stay competitive. • Due to the new market drivers mentioned above, we have come to believe it is likely just a matter of a year or two before 120 fps cameras are available at our target price of ca. $200/camera. This is in part because of the large markets involved and in part because of a gaming/cell phone business model that subsidizes hardware financially. •
Appendix A –Selected Imaging System Website References
Imaging Chips and Design Information: OmniVision Camera-Cube Technology: http://www.ovt.com/products/category.php?id=21
Article About New OmniVision chip That Offers VGA Resolution at 120 fps: OmniVision creates world's smallest 1080p sensor OmniVision Imaging Chip Supplier Website: OmniVision HD Reference Design from eInfoChips: http://www.image-acquire.com/hd-ip-camera-reference-design/ Appro Photoelectron Inc. - Reference System Design Company: http://www.appropho.com/NewWeb/Product_All.php Link Discussing T.I. Reference Design Info.: http://www.soccentral.com/PrintPage.asp?PassedEntryID=26856 T.I. Camera Reference Designs (Some Promising $150 BoM Costs): http://focus.ti.com/apps/docs/mrktgenpage.tsp?contentId=41246&appId=79 Analog Devices “Smart Camera” Reference Design Press Release: http://www.analog.com/en/pressrelease/3_30_09_ADI_Speeds_Smart_Camera_Development/press.html eInfoChips IP Camera Reference Design: DM365IPNCVA-MT5 : Advanced Video Analytics (VA) DM365 IPCamera Reference Design - www.einfochips.com Image Processing Chip Supplier: Stream Processors Inc.: Reference Design Kits DigiLink Software – Firmware for video camera designs: http://www.digilinksoftware.com/main/products.html Article About Designing Imaging Systems with Good Dynamic Range: V ideo/Imaging DesignLine | Building a video camera th a t works in all lighting conditions, 24/7
High-Dynamic Range Imaging: http://www.pixim.com/products-and-technology/technology Article about T.I. Image Data Encoding Chip: New TMS320DM368 DaVinci video processor from Texas Instruments Incorporated extends the range of its DM36x generation portable encoding
High-Speed Camera Suppliers: High-Speed Imaging Camera Supplier: http://www.siliconimaging.com/index1.htm “Red Cameras” – Supplier of High-Speed Video Cameras: RED / FAQ OMNIVISION AND EASIC OFFER MPEG-4 REFERENCE DESIGN TO PROVIDE RAPID DEVELOPMENT OF CAMERA SOLUTIONS
Point Grey Research – High-Speed Camera Canadian Favourites: http://www.ptgrey.com/
Project Natal and Sony “Move”: Project Natal Rumours - $149 Price: h ttp://www.techgeeze.com/2010/05/project-nata l- to-cost-149-standalone.html
Project Natal – More Info./Rumours Re: June 13, 2010 Announcement Date – Could It All Be True? http://www.techgeeze.com/2010/03/xbox-project-natal-to-unveil-on-june.html Sony PlayStation “Move” – General Information: http://us.playstation.com/ps3/playstation-move/ Sony “Move” – More Information: h ttp://www.mcvuk.com/news/39200/Sony-Move-is-not-Wii-on-PS3
Interesting “Big-Picture” Articles: Article About Creation of Open-Source Camera Hardware Platform at Stanford: http://news.stanford.edu/news/2009/august31/levoy-opensource-camera090109.html
