This post will present the experiment – assessing a sample of open source predictions for the year 2012 – address the methodological problems encountered while creating the evaluation itself, and underline the lessons learned. The second part (forthcoming) will discuss results.