Some more interesting reading:
K. Price, Anything You Can Do, I Can Do Better (No You Can’t)…, Computer Vision, Graphics, and Image Processing, Vol. 36, pp. 387-391, 1986, 
doi:10.1016/0734-189X(86)90083-6.
Abstract: Computer vision suffers from an overload of written information but a dearth of good evaluations and comparisons. This paper discusses why some of the problems arise and offers some guidelines we should all follow.
Very nice reading material, and (although I know these ideas are around for quite some time already) I was amazed to see so many parallels to our recent IEEE Signal Processing Magazine paper, already in this paper by Price from 1986. That’s more than 20 years ago! Price talks about the reproducibility problems in computer vision and image processing, writing we should “stand on other’s shoulders, not on other’s toes”. He also did a study on reproducibility of a set of about 42 papers, verifying the size of the dataset and clarity of the problem statement. Price concludes as follows: “Researchers should make the effort to obtain implementations of other researchers’ systems so that we can better understand the limitations of our own work.”
Again, interesting to see how these issues and worries have been around for more than 20 years in the field of image processing. It’s about time to drastically improve our standards, I think!
I would really recommend this article to anyone interested in issues around reproducible research.
Impressive. Very impressive.
As you can see, I am again impressed by the annual SIGGRAPH conference that took place last August, and about which my colleagues reported. There were more than 28000 participants, and the acceptance ratio for the presented papers was below 20%. While the main focus of the conference is on computer graphics, it also includes a wide range of presentations on 3D, image and video enhancement, and image processing in general. Next to these technical sessions, there are also movie screenings, and a computer animation festival.
But, apart from the high quality and interesting mix of topics, I also really like the way papers are presented. Certainly for people like me, who did not attend the conference. Each paper starts off (after the title and author list) with a “telling illustration”, graphically illustrating the paper. Really nice to get a quick idea about the paper. Moreover, for most of those papers, the authors also have a nice video presenting their paper on their website. I have no idea whether that is mandatory, and whether one could find all those presentation videos on the ACM website. My colleagues also told me that all the presentations from this year’s SIGGRAPH conference would be recorded and made available online. I am curious! It’s still not the same as actually going there, but it is as close as I can get. For now.
One of the reproducibility problems with many current papers is that everyone applies his new algorithm to his own set of data. So did I in my super-resolution work, too. A problem with that is that it is very difficult to assess whether the data set is used (a) because that was the one the author had at hand, (b) because it was the most representative one, or (c) because the algorithm performed best on that data set.
To allow more fair comparisons, competitions are being set up in various fields. Often in the period before a conference, a competition is set up, where everyone can try his algorithm on a common dataset given by the organizers.
Continue reading ‘Data set competitions’
An article close to my current work on 3D now:
D. Scharstein and R. Szeliski, A taxonomy and evaluation of dense two-frame stereo correspondence algorithms, International Journal of Computer Vision, 47(1/2/3), pp. 7-42, April-June 2002.
In their article, Scharstein and Szeliski make a comparison of stereo estimation algorithms. But they do not just offer this overview of algorithms. On their webpage, they also provide the source code, and a widely used dataset of stereo images. They also invite other researchers to try their own algorithm on this dataset, and upload the results. This has resulted over the years in a performance comparison of almost 50 stereo algorithms, nicely listed on their webpage.
A nice example of what reproducible research can do! I think we need a lot more of these comparisons on common (representative) datasets.
For years, camera manufacturers have been building cameras with more and more pixels. It seems/seemed as if there is no limit to the number of megapixels you can get in your camera.
In theory, this is true: the more pixels you have in your camera, the more details you can capture, and the larger the prints you can make. However, we also all want small, compact cameras. So the camera manufacturers do their best to fit this larger number of pixels into the same or an even smaller camera housing, with the same sensor and lens size. And that’s where it gets tricky, because if you increase the number of pixels and keep the sensor size the same, the size of each pixel has to decrease. And a smaller pixel can capture less light, which results in relatively more noise. So the resulting images are noisier. Also, it does not make much sense to have a sensor that can capture more details if your lens does not let those details pass through. The lens should therefore also be replaced by one with a higher quality, and better lenses are typically also bigger.
So, there is an optimum. And the people at Image Engineering have measured that the optimum for a compact camera is 6 megapixels. They have even devoted a website to it. Of course, this does not hold if you also increase the sensor size (for example in SLR cameras). It is therefore better to say that the optimal pixel size is 3 micrometer. Depending on the sensor size, one can then compute the optimal number of pixels.
The digital camera manufacturer Leica built a pixelated dog to promote their new camera (Leica D-Lux 3), using small blocks, glue, and a lot of time. Or, as they put it, to show what the world would look like with less pixels.
Strange effect, such a pixelated object in a real world!
(done by marketing firm Philipp und Keuntje, and seen at DS Marketingblog and Gizmodo)
