Report

Teaser, summary, work performed and final results

Periodic Reporting for period 1 - Virtual Optics (Virtual Optics - A software revolution in the optical industry)

Teaser

Intelligent Imaging Solutions GmbH (IIS) is a company offering software solutions in the field of computational photography. Over the past few years, it has successfully marketed software solutions for end-consumers in the photography market. The software solutions were highly...

Summary

Intelligent Imaging Solutions GmbH (IIS) is a company offering software solutions in the field of computational photography. Over the past few years, it has successfully marketed software solutions for end-consumers in the photography market. The software solutions were highly specialized and the first of their kind. These solutions received a lot of attention and positive feedback from both customers and the media.

The products are aimed towards consumers in the digital-camera market. Even though cameras and lenses in this market can cost several thousand euros, the full potential of the technology has not been fully unlocked so far. The feasibility study completed under the Horizon2020 scheme had two goals to change this:

1) To demonstrate that modern computers with powerful graphic cards (GPUs) can be used to process and enhance high-resolution aerial photos within a reasonable time.
2) To evaluate the market potential for a software solution capable of correcting complex optical aberrations in the aerial-camera market.

Camera lenses are a critical component of optical-imaging systems, and lens imperfections compromise image quality. Manufacturers of photographic lenses attempt to minimize optical aberrations by combining several glass elements in so-called compound lenses comprising as many as fifteen elements or more. As a result, high-grade lenses are probably the most expensive components of high-end camera systems today and specialized lenses for aerial photography can cost tens of thousands of Euros. Optical aberrations are inevitable and the design of a lens is always a trade-off between various parameters, including price. In fact, it is impossible to make a perfect lens, and it is very expensive to make a close-to-perfect lens. This has not been a big concern over the past decades in photography as the optical quality of lenses easily outperformed the sensor / film resolution. However, this has changed recently due to the ever increasing resolution of modern camera sensors. Optical aberrations have now become a problem for image quality—and it becomes increasingly expensive to correct optical aberrations, particularly for challenging lighting situations. If one wanted to improve the optical performance of lenses for high-resolution aerial cameras further, the costs would increase by 200–500 percent which would make these cameras economically unfeasible.

A much cheaper solution can be found within the new field of computational photography. IIS is currently the only company capable of correcting spatially-varying complex optical aberrations. This means that IIS can significantly improve the sharpness of images from high-resolution sensors that suffer from a decrease in sharpness towards the edges. The process of reverting image degradations stemming from imperfect optics by computational means is known as image deconvolution.

There are several reasons why correcting optical aberrations by the means of image deconvolution have not been widely used so far:
• The optical aberrations need to be known and described in a mathematically complex way. Determining these aberrations is very labor intensive and they vary significantly for different aperture and focus settings. Lens-to-lens variation is another problem making it almost impossible to leverage the technology on a larger scale particularly in the consumer sector.
• Reversing optical aberrations is computationally very expensive, especially for large images. The information of several hundreds (or even thousands) of pixels in the neighboring region has to be taken into account in order to recover a single pixel. Recovering an image of 60–200 MP in size is computationally challenging and only possible on high-performance computers. So far, the computation time for a single image was in the order of tens of minutes which is too long for the field of aerial photography, where typically hundreds to thousands of images are taken during a single flight.

Work performed

The feasibility study and the evaluation of the economic potential comprised several work packages:

• Optimize the critical parts of the base code (technology) for GPUs and thus achieve a significant increase in speed:
o The core algorithm for reversing optical aberrations was successfully (in parts) ported to GPU code. The speed increase was close to 80 percent—significantly better than expected. The prototype, therefore, proved the feasibility of the technology in the image-processing pipeline with respect to processing times.
o The initial idea to determine the optical aberrations “blindly” (without calibration data) was abandoned as the improvements were not as good as expected and retrieving calibration data from the lens can be performed in a cost efficient way for a small number of lenses in the aerial-imaging sector. For the mass consumer market this is not possible as the costs for determining the measurement data is higher than the cost of producing a consumer lens.
o The objective to optimize the crucial parts of the GPU code was, therefore, successfully fulfilled. The technological approach was adopted to a “non-blind” correction of the optical aberrations after some coordination and initial tests with an aerial camera manufacturer.

• Evaluate the market potential for aerial-camera systems
o The market potential that was initially envisioned for this technology was confirmed.
o The best strategy to serve the market was through manufacturers of aerial-camera systems. The business model, therefore, focuses on licensing the technology.

• Finalize market potential and determine pricing of technology to end customers (B2B) and licensing partners:
o Possible partners were identified and the technology was demonstrated

The feasibility study confirmed the market potential, pricing, and the technology. Several adjustments to the “go to market” strategy as well as to the technological approach were made throughout the feasibility study.

Final results

The feasibility study confirmed that IIS is currently the technological market leader for correcting optical aberrations in high-resolution cameras. It was demonstrated that the algorithms can be speeded up significantly for large images leveraging GPUs—thus opening up new possibilities that were not thought of before.

The potential impact could be far reaching as the technology can be considered disruptive in the market for aerial cameras. However, to exploit the full potential, a strong market player has to be won over. Integrating the algorithms in the image-processing pipeline could further strengthen Europe’s position as the leader for aerial cameras and protect jobs directly and indirectly against competition from abroad. Over the past years, technological advancements in optics from Chinese and Korean companies posed a significant threat to the existing European players. Whereas the optical and chip industry has mainly moved to the East, software development for crucial components is still done in Europe. If the share of value creation achieved by means of software is increased, so is the sustainable competitive advantage and ultimately job security in Europe.

Website & more info

More info: http://www.virtual-optics.de.