MiddEvalSDK Software development kit for the Middlebury Stereo Evaluation v.3 Version 1.6 Daniel Scharstein September 14, 2015 =============================== Contents ================================== README.txt -- this file CHANGES.txt -- change log COPYING_LESSER.txt, COPYING.txt -- license (GNU Lesser General Public License) alg-ELAS/ -- sample stereo algorithm: Libelas by A. Geiger, adpated with permission from http://www.cvlibs.net/software/libelas/ alg-ELAS/run -- sample run script (adapt this for your algorithm) code/ -- C++ code used by the scripts runalg -- script to run algorithm on datasets runeval -- script to evaluate algorithm results runevalF -- script to evaluate algorithm results with full-size GT runviz -- script to visualize .pfm disparities as .png makezip -- script to create zip archive of results cleanalg -- script to remove files created by algorithms In addition to the SDK files, you will also need to download input data and ground truth data, and unzip at the same level as the SDK files. This should result in a subset of the following directories: testF/ testH/ testQ/ trainingF/ trainingH/ trainingQ/ ============================= Quick Start ================================= Follow these steps to get started. We'll use the quarter-resolution files for now. 1. Download and unzip MiddEval3-data-Q.zip and MiddEval3-GT0-Q.zip. You should have a directory MiddEval3 with the following contents: COPYING.txt README.txt cleanalg makezip runeval runviz testQ/ COPYING_LESSER.txt alg-ELAS/ code/ runalg runevalF trainingQ/ 2. Compile Libelas as follows: cd alg-ELAS/build cmake .. make cd ../.. 3. Compile the tools in code/ as follows: cd code/imageLib make cd .. make cd .. 4. Run ELAS: ./runalg -- shows usage ./runalg Q Motorcycle ELAS -- run ELAS on one dataset ./runalg Q Motorcycle ELAS v2 -- run again, save results with suffix 'v2' ./runalg Q training ELAS -- run ELAS on all 15 training sets 5. Evaluate results by ELAS: ./runeval -- shows usage ./runeval Q Motorcycle 1 ELAS -- evaluate ELAS with error thresh t=1.0 ./runeval Q Motorcycle 2 ELAS_s -- evaluate "sparse" ELAS results, t=2.0 ./runeval Q Motorcycle 1 -- evaluate all results for Motorcycle, t=1.0 ./runeval Q training 0.5 ELAS -- evaluate ELAS on all training sets, t=0.5 *** NOTE: For efficiency, the evaluation script 'runeval' uses the GT file disp0GT.pfm with the SAME resolution as your disparity map. In contrast, the "official" online evaluation evaluates at FULL resolution, and upsamples your results if necessary. Error thresholds need to be converted accordingly, e.g., a threshold of 2.0 at full resolution would correspond to a threshold of 0.5 at quarter resolution. Even with this conversion, the numbers differ slightly when evaluating with the GT at full resolution. To get the official numbers, you can use the script 'runevalF'. You'll need to download the full-size files MiddEval3-data-F.zip and MiddEval3-GT0-F.zip in order to use this script. Once you do, try ./runevalF Q Motorcycle 1.0 ELAS -- official numbers using full-res GT ./runeval Q Motorcycle 0.25 ELAS -- approximate numbers using Q-size GT In the online table for the training sets, the name of each method links to a zip file of the results. You can download such a zip file, say results_SGM_Q.zip, unzip it *within* MiddEval3, and evaluate these results as well: ./runeval Q training 1 ELAS SGM Finally, to get evaluation output in column format ready to be imported into other programs such as Excel or gnuplot, use option -b ("brief"): ./runevalF -b Q all 2.0 ./runeval -b Q all 0.5 6. Examine disparity maps We highly recommend to install Heiko Hirschmueller's "cvkit" (link from webpage), which includes a powerful image viewer "sv". Once installed, try the following: sv trainingQ/Motorcycle/disp*pfm Hit 'h' for help, 'm' to change colormaps, 'k' to keep the colormap setting, and the arrow keys to switch between files. Most amazing: hit 'p' to start the 3D viewer 'plyv' and visualize the disparities in 3D! (plyv reads the camera parameters from the calib.txt files.) Alternatively, run ./runviz Q to translate all disp*pfm files into disp*-c.png color encodings, and view those using any image viewer. On subsequent runs, runviz will only re-convert any pfm files that have changed. 7. Add your own algorithm: Create a directory alg-XXX, where XXX is a (short) name for your method. Then create a "run" script within your alg-XXX directory, using the one in alg-ELAS as a model. Once done, test your algorithm. For example, calling the new method "Ours": ./runalg Q Motorcycle Ours ./runeval Q Motorcycle 1 Ours A few words on the disparity encoding: We use floating-point disparities in PFM format, with INFINITY representing unknown values. If your algorithm produces disparities in other formats (e.g., scaled 16-bit PNG or floating-point TIF), that's ok, and you can convert the output using the "imgcmd" tool (part of cvkit). Alternately, you can use the tool code/disp2pfm to convert integer disparities (e.g. in pgm format) to pfm. See the alg-ELAS/run script for more detailed information. Note, however, that many of the half-size datasets and most of the full-size datasets have disparity ranges that exceed 255. Thus, if your method produces 8-bit integer disparities, you are limited to working with the quarter-resolution datasets and won't be able to achieve accurate results when evaluated at full resolution. Alternately, you can change your implementation to write PFM disparities directly. You can use the following code as examples: alg-ELAS/src/main.cpp code/imageLib/ImageIO.cpp 8. Submit your results: Our online web interface supports uploading of zip files of results on either just the 15 training images or all 30 training and test images. You can upload the results on just the training set in order to see a temporary table comparing your results with those uploaded by others. To do this, run ./makezip Q training Ours and upload the resulting file resultsQ-Ours.zip using the web interface. This scripts ensures that all necessary files are present and that the runtimes files contain numbers. Once you are ready to submit your results to the permanent table, you must upload a complete set of results on ALL 30 datasets (15 training, 15 test). All images must have the same resolution (one of Q, H, or F). You must use constant parameter settings (except for disaprity ranges), and you cannot "mix and match" resolutions (e.g., use trainingQ/Piano and trainingH/Teddy). All of this is ensured if you use "runalg all", e.g.: ./runalg Q all Ours Then, create a zip file with all results using ./makezip Q all Ours and upload the resulting file using the web interface. You will then be able to provide information about your method and request publication of both training and test results. Anonynous listings for double-blind review processes are possible. Note that in order to prevent parameter tuning to the test data, you will have only one shot at evaluating your results on the test data. In other words, you may not request publication for two different sets of results by the same method. Also, your results on the training data will be available for download by others. So please be sure to only submit your final set of results for publication. You may of course upload and evaluate results on the training set as often as you want. 9. To clean results created by algs (disp*pfm, time*txt, disp*-c.png), use: ./cleanalg -- shows usage ./cleanalg Q ELAS -- removes all results by ELAS ./cleanalg Q ELASv2 -- removes all results by ELASv2 ./cleanalg Q ELAS\* -- removes all results by ELAS (all versions)