vision.middlebury.edu/stereo/eval

Middlebury Stereo Evaluation - Version 2

New features and main differences to version 1.
Submit and evaluate your own results.

Error Threshold = 1		Sort by nonocc				Sort by all				Sort by disc

Algorithm	Avg.	Tsukuba ground truth			Venus ground truth			Teddy ground truth			Cones ground truth			Average percent of bad pixels (explanation)
	Rank	nonocc	all	disc	nonocc	all	disc	nonocc	all	disc	nonocc	all	disc

CooptRegion [41]	3.3	0.87 1	1.16 1	4.61 1	0.11 2	0.21 2	1.54 3	5.16 8	8.31 4	13.0 7	2.79 4	7.18 1	8.01 6	4.41
AdaptingBP [17]	3.5	1.11 8	1.37 4	5.79 9	0.10 1	0.21 3	1.44 1	4.22 4	7.06 2	11.8 4	2.48 1	7.92 3	7.32 2	4.23
DoubleBP2 [35]	4.1	0.88 3	1.29 2	4.76 3	0.13 4	0.45 7	1.87 6	3.53 2	8.30 3	9.63 1	2.90 5	8.78 10	7.79 3	4.19
OutlierConf [42]	5.2	0.88 2	1.43 6	4.74 2	0.18 9	0.26 4	2.40 11	5.01 6	9.12 7	12.8 6	2.78 3	8.57 6	6.99 1	4.60
DoubleBP [15]	6.2	0.88 4	1.29 3	4.76 4	0.14 6	0.60 15	2.00 8	3.55 3	8.71 6	9.70 2	2.90 6	9.24 13	7.80 4	4.30
SubPixDoubleBP [30]	6.9	1.24 12	1.76 15	5.98 10	0.12 3	0.46 8	1.74 5	3.45 1	8.38 5	10.0 3	2.93 7	8.73 9	7.91 5	4.39
AdaptOvrSegBP [33]	11.5	1.69 24	2.04 23	5.64 8	0.14 5	0.20 1	1.47 2	7.04 16	11.1 9	16.4 13	3.60 13	8.96 12	8.84 12	5.59
SymBP+occ [7]	12.7	0.97 6	1.75 14	5.09 6	0.16 7	0.33 5	2.19 9	6.47 10	10.7 8	17.0 16	4.79 26	10.7 23	10.9 22	5.92
PlaneFitBP [32]	12.7	0.97 7	1.83 16	5.26 7	0.17 8	0.51 10	1.71 4	6.65 11	12.1 15	14.7 9	4.17 22	10.7 22	10.6 21	5.78
AdaptDispCalib [36]	13.7	1.19 10	1.42 5	6.15 11	0.23 11	0.34 6	2.50 13	7.80 21	13.6 23	17.3 19	3.62 14	9.33 14	9.72 17	6.10
Segm+visib [4]	14.1	1.30 17	1.57 7	6.92 21	0.79 24	1.06 21	6.76 26	5.00 5	6.54 1	12.3 5	3.72 15	8.62 8	10.2 19	5.40
C-SemiGlob [19]	14.5	2.61 33	3.29 28	9.89 31	0.25 14	0.57 12	3.24 18	5.14 7	11.8 10	13.0 7	2.77 2	8.35 5	8.20 7	5.76
SO+borders [29]	14.8	1.29 16	1.71 11	6.83 18	0.25 15	0.53 11	2.26 10	7.02 15	12.2 16	16.3 11	3.90 17	9.85 18	10.2 20	6.03
DistinctSM [27]	16.1	1.21 11	1.75 13	6.39 13	0.35 16	0.69 18	2.63 15	7.45 20	13.0 19	18.1 21	3.91 18	9.91 20	8.32 9	6.14
OverSegmBP [26]	16.8	1.69 25	1.97 20	8.47 27	0.51 20	0.68 17	4.69 22	6.74 12	11.9 14	15.8 10	3.19 10	8.81 11	8.89 13	6.11
CostAggr+occ [39]	16.8	1.38 19	1.96 19	7.14 22	0.44 18	1.13 23	4.87 23	6.80 13	11.9 12	17.3 18	3.60 12	8.57 7	9.36 15	6.20
SegmentSupport [28]	17.1	1.25 13	1.62 9	6.68 15	0.25 13	0.64 16	2.59 14	8.43 26	14.2 24	18.2 22	3.77 16	9.87 19	9.77 18	6.44
RegionTreeDP [18]	17.8	1.39 21	1.64 10	6.85 19	0.22 10	0.57 12	1.93 7	7.42 19	11.9 13	16.8 15	6.31 34	11.9 29	11.8 25	6.56
EnhancedBP [24]	18.8	0.94 5	1.74 12	5.05 5	0.35 17	0.86 19	4.34 21	8.11 24	13.3 21	18.5 25	5.09 29	11.1 25	11.0 23	6.69
SegTreeDP [22]	19.9	2.21 30	2.76 25	10.3 33	0.46 19	0.60 14	2.44 12	9.58 29	15.2 29	18.4 24	3.23 11	7.86 2	8.83 11	6.82
AdaptWeight [12]	20.2	1.38 19	1.85 17	6.90 20	0.71 22	1.19 24	6.13 24	7.88 22	13.3 22	18.6 27	3.97 21	9.79 17	8.26 8	6.67
InteriorPtLP [34]	20.2	1.27 14	1.62 8	6.82 17	1.15 28	1.67 27	12.7 36	8.07 23	11.9 11	18.7 28	3.92 20	9.68 15	9.62 16	7.26
ImproveSubPix [25]	21.0	3.00 35	3.61 32	10.9 35	0.88 26	1.47 25	7.10 28	7.12 17	12.4 18	16.6 14	2.96 8	8.22 4	8.55 10	6.90
SemiGlob [6]	22.4	3.26 36	3.96 33	12.8 38	1.00 27	1.57 26	11.3 32	6.02 9	12.2 17	16.3 12	3.06 9	9.75 16	8.90 14	7.50
VariableCross [44]	25.2	1.99 27	2.65 24	6.77 16	0.62 21	0.96 20	3.20 17	9.75 30	15.1 28	18.2 23	6.28 33	12.7 31	12.9 32	7.60
RealtimeBP [21]	25.6	1.49 22	3.40 30	7.87 25	0.77 23	1.90 30	9.00 31	8.72 28	13.2 20	17.2 17	4.61 24	11.6 27	12.4 30	7.69
2OP+occ [37]	25.8	2.91 34	3.56 31	7.33 24	0.24 12	0.49 9	2.76 16	10.9 35	15.4 30	20.6 32	5.42 31	10.8 24	12.5 31	7.75
GC+occ [2]	27.3	1.19 9	2.01 22	6.24 12	1.64 33	2.19 32	6.75 25	11.2 36	17.4 35	19.8 31	5.36 30	12.4 30	13.0 33	8.26
Layered [5]	27.9	1.57 23	1.87 18	8.28 26	1.34 30	1.85 28	6.85 27	8.64 27	14.3 25	18.5 26	6.59 36	14.7 34	14.4 35	8.24
MultiCamGC [3]	27.9	1.27 15	1.99 21	6.48 14	2.79 39	3.13 36	3.60 20	12.0 37	17.6 36	22.0 35	4.89 27	11.8 28	12.1 27	8.31
AdaptPolygon [43]	29.2	2.29 31	2.88 27	8.94 29	0.80 25	1.11 22	3.41 19	10.5 34	15.9 31	21.3 34	6.13 32	13.2 32	13.3 34	8.32
GenModel [20]	30.7	2.57 32	4.74 36	13.0 39	1.72 34	3.08 35	16.9 38	6.86 14	15.0 27	19.2 29	4.64 25	14.9 35	11.4 24	9.50
TensorVoting [9]	31.1	3.79 37	4.79 37	8.86 28	1.23 29	1.88 29	11.5 33	9.76 31	17.0 34	24.0 38	4.38 23	11.4 26	12.2 28	9.25
RealTimeGPU [14]	31.4	2.05 29	4.22 35	10.6 34	1.92 36	2.98 34	20.3 40	7.23 18	14.4 26	17.6 20	6.41 35	13.7 33	16.5 37	9.82
CostRelax [11]	32.4	4.76 42	6.08 40	20.3 44	1.41 32	2.48 33	18.5 39	8.18 25	15.9 32	23.8 36	3.91 19	10.2 21	11.8 26	10.6
ReliabilityDP [13]	33.6	1.36 18	3.39 29	7.25 23	2.35 38	3.48 39	12.2 35	9.82 32	16.9 33	19.5 30	12.9 44	19.9 43	19.7 39	10.7
TreeDP [8]	34.8	1.99 28	2.84 26	9.96 32	1.41 31	2.10 31	7.74 29	15.9 41	23.9 41	27.1 42	10.0 40	18.3 39	18.9 38	11.7
GC [1d]	35.7	1.94 26	4.12 34	9.39 30	1.79 35	3.44 38	8.75 30	16.5 42	25.0 43	24.9 39	7.70 37	18.2 38	15.3 36	11.4
BP+MLH [40]	36.1	4.17 40	6.34 41	14.6 41	1.96 37	3.31 37	16.8 37	10.2 33	18.9 37	24.0 37	4.93 28	15.5 36	12.3 29	11.1
DP [1b]	39.8	4.12 39	5.04 38	12.0 36	10.1 47	11.0 47	21.0 41	14.0 38	21.6 38	20.6 32	10.5 41	19.1 40	21.1 41	14.2
PhaseBased [31]	41.7	4.26 41	6.53 42	15.4 42	6.71 42	8.16 42	26.4 44	14.5 39	23.1 39	25.5 40	10.8 43	20.5 44	21.2 42	15.3
RegionalSup [38]	41.9	3.99 38	6.05 39	14.2 40	8.14 43	9.68 44	36.8 46	18.3 45	26.7 45	32.1 44	9.16 38	19.3 41	19.9 40	17.0
SSD+MF [1a]	42.1	5.23 45	7.07 43	24.1 45	3.74 40	5.16 40	11.9 34	16.5 43	24.8 42	32.9 45	10.6 42	19.8 42	26.3 44	15.7
STICA [16]	43.4	7.70 46	9.63 47	27.8 46	8.19 44	9.58 43	40.3 47	15.8 40	23.2 40	37.7 46	9.80 39	17.8 37	28.7 46	19.7
SO [1c]	44.0	5.08 44	7.22 45	12.2 37	9.44 46	10.9 46	21.9 42	19.9 46	28.2 47	26.3 41	13.0 45	22.8 46	22.3 43	16.6
PhaseDiff [23]	44.8	4.89 43	7.11 44	16.3 43	8.34 45	9.76 45	26.0 43	20.0 47	28.0 46	29.0 43	19.8 47	28.5 47	27.5 45	18.8
Infection [10]	45.0	7.95 47	9.54 46	28.9 47	4.41 41	5.53 41	31.7 45	17.7 44	25.1 44	44.4 47	14.3 46	21.3 45	38.0 47	20.7

References

[1]	D. Scharstein and R. Szeliski. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. IJCV 2002. a - SSD + min-filter (i.e., shiftable windows), window size = 21 b - Dynamic programming, similar to Bobick and Intille (IJCV 1999) c - Scanline optimization (1D optimization using horizontal smoothness terms) d - Graph cuts using alpha-beta swaps (Boykov, Veksler, and Zabih, PAMI 2001)
[2]	V. Kolmogorov and R. Zabih. Computing visual correspondence with occlusions using graph cuts. ICCV 2001.
[3]	V. Kolmogorov and R. Zabih. Multi-camera scene reconstruction via graph cuts. ECCV 2002.
[4]	M. Bleyer and M. Gelautz. A layered stereo algorithm using image segmentation and global visibility constraints. ICIP 2004.
[5]	L. Zitnick, S.B. Kang, M. Uyttendaele, S. Winder, and R. Szeliski. High-quality video view interpolation using a layered representation. SIGGRAPH 2004.
[6]	H. Hirschmüller. Accurate and efficient stereo processing by semi-global matching and mutual information. CVPR 2005, PAMI 30(2):328-341, 2008.
[7]	J. Sun, Y. Li, S.B. Kang, and H.-Y. Shum. Symmetric stereo matching for occlusion handling. CVPR 2005.
[8]	O. Veksler. Stereo correspondence by dynamic programming on a tree. CVPR 2005.
[9]	P. Mordohai and G. Medioni. Stereo using monocular cues within the tensor voting framework. PAMI 28(6):968-982, 2006.
[10]	G. Olague, F. Fern�ndez, C. P�rez, and E. Lutton. The infection algorithm: an artificial epidemic approach for dense stereo correspondence. Artificial Life, 2006.
[11]	R. Brockers, M. Hund, and B. Mertsching. Stereo vision using cost-relaxation with 3D support regions. Image and Vision Computing New Zealand (IVCNZ), 2005.
[12]	K.-J. Yoon and I.-S. Kweon. Adaptive support-weight approach for correspondence search. PAMI 28(4):650-656, 2006.
[13]	M. Gong and Y.-H. Yang. Near real-time reliable stereo matching using programmable graphics hardware. CVPR 2005.
[14]	L. Wang, M. Liao, M. Gong, R. Yang, and D. Nist�r. High-quality real-time stereo using adaptive cost aggregation and dynamic programming. 3DPVT 2006.
[15]	Q. Y�ng, L. Wang, R. Yang, H. Stew�nius, and D. Nist�r. Stereo matching with color-weighted correlation, hierarchical belief propagation and occlusion handling. CVPR 2006.
[16]	H. Audirac, A. Beloiarov, F. N��ez, and J. Villegas. Dense disparity map based on STICA algorithm. Expo Forestal, Mexico, 2005.
[17]	A. Klaus, M. Sormann and K. Karner. Segment-based stereo matching using belief propagation and a self-adapting dissimilarity measure. ICPR 2006.
[18]	C. Lei, J. Selzer, and Y. Yang. Region-tree based stereo using dynamic programming optimization. CVPR 2006.
[19]	H. Hirschmüller. Stereo vision in structured environments by consistent semi-global matching. CVPR 2006, PAMI 30(2):328-341, 2008.
[20]	C. Strecha, R. Fransens, and L. Van Gool. Combined depth and outlier estimation in multi-view stereo. CVPR 2006.
[21]	Q. Yang, L. Wang, R. Yang, S. Wang, M. Liao, and D. Nist�r. Real-time global stereo matching using hierarchical belief propagation. BMVC 2006.
[22]	Y. Deng and X. Lin. A fast line segment based dense stereo algorithm using tree dynamic programming. ECCV 2006.
[23]	S. El-Etriby, A. Al-Hamadi, and B. Michaelis. Dense depth map reconstruction by phase difference-based algorithm under influence of perspective distortion. ICCVG 2006 / J. Machine Graphics and Vision.
[24]	S. Larsen, P. Mordohai, M. Pollefeys, and H. Fuchs. Temporally consistent reconstruction from multiple video streams using enhanced belief propagation. ICCV 2007.
[25]	S. Gehrig and U. Franke. Improving sub-pixel accuracy for long range stereo. ICCV VRML workshop 2007.
[26]	L. Zitnick and S.B. Kang. Stereo for image-based rendering using image over-segmentation. IJCV 2007.
[27]	K.-J. Yoon and I. S. Kweon. Stereo matching with the distinctive similarity measure. ICCV 2007.
[28]	F. Tombari, S. Mattoccia, and L. Di Stefano. Segmentation-based adaptive support for accurate stereo correspondence. PSIVT 2007.
[29]	S. Mattoccia, F. Tombari, and L. Di Stefano. Stereo vision enabling precise border localization within a scanline optimization framework. ACCV 2007.
[30]	Q. Yang, R. Yang, J. Davis, and D. Nist�r. Spatial-depth super resolution for range images. CVPR 2007.
[31]	S. El-Etriby, A. Al-Hamadi, and B. Michaelis. Dense stereo correspondence with slanted surface using phase-based algorithm. IEEE ISIE 2007.
[32]	Anonymous. Near real-time stereo for weakly-textured scenes. Submitted to CVPR 2008.
[33]	Y. Taguchi, B. Wilburn, and L. Zitnick. Stereo reconstruction with mixed pixels using adaptive over-segmentation. CVPR 2008.
[34]	A. Bhusnurmath and C.J. Taylor. Solving stereo matching problems using interior point methods. 3DPVT 2008.
[35]	Q. Yang, L. Wang, R. Yang, H. Stew�nius, and D. Nist�r. Stereo matching with color-weighted correlation, hierarchical belief propagation and occlusion handling. Submitted to PAMI.
[36]	Z.Gu, X.Su, Y.Liu, and Q.Zhang. Local stereo matching with adaptive support-weight, rank transform and disparity calibration. Pattern Recognition Letters, 2008.
[37]	O. Woodford, P. Torr, I. Reid, and A. Fitzgibbon. Global stereo reconstruction under second order smoothness priors. CVPR 2008.
[38]	M. Stivanello, E. Leal, N. Palluat, et al. Stereo vision based on local correspondence with regional support. Submitted to Image and Vision Computing.
[39]	D. Min and K. Sohn. Cost aggregation and occlusion handling with WLS in stereo matching. IEEE TIP 2008.
[40]	O. Stankiewicz and K. Wegner. Depth map estimation software version 2. ISO/IEC MPEG meeting, Archamps, France, 2008.
[41]	Z. Wang and Z. Zheng. A region based stereo matching algorithm using cooperative optimization. CVPR 2008.
[42]	L. Xu and J. Jia. Stereo matching: an outlier confidence approach. ECCV 2008.
[43]	J. Lu, G. Lafruit, and F. Catthoor. Anisotropic local high-confidence voting for accurate stereo correspondence. Proc. SPIE, vol. 6812, 2008.
[44]	K. Zhang, J. Lu, and G. Lafruit. Cross-based local stereo matching using orthogonal integral images. Submitted to IEEE TCSVT 2008.

This page was designed by Daniel Scharstein and Anna Blasiak. Please send feedback and bug reports to schar@middlebury.edu.
Support for this work was provided in part by NSF grant 0413169. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.