[ncdnhc-discuss] The NCDNHC's .org report is numerically inconsistent.

[Thomas Roessler]

First of all, I'd like to congratulate the NCDNHC team for the  
great amount of work spent on its report for the .org bid 
evaluation.


However, the numerical material provided in the report is  
inconsistent.

The most obvious problem occurs in the table on page 49, where 
responsiveness scores are simply sorted in decreasing order.  Here's 
a corrected version of that table:

+-------------+----------------+---------+-----------------+---------+
| name        | responsiveness | support | differentiation | total   |
+-------------+----------------+---------+-----------------+---------+
| unity       |          27.25 |       9 |            20.5 | 24.5575 |
| isoc        |          21.75 |       9 |            14.5 | 20.6725 |
| ims/isc     |             14 |       7 |              15 |   16.78 |
| gnr         |          26.75 |       3 |              14 | 15.8225 |
| uia         |          16.75 |       5 |             7.5 | 12.5225 |
| neustar     |          12.75 |       3 |              15 | 12.4425 |
| dotorg      |           20.5 |       1 |               9 |  10.135 |
| registerorg |          11.75 |       0 |              16 |  9.5725 |
| .org        |              5 |       5 |               5 |    8.35 |
| switch      |              8 |       0 |              10 |    6.16 |
| organic     |              0 |       0 |            11.5 |     4.6 |
+-------------+----------------+---------+-----------------+---------+

(total = 0.27 * responsiveness + support + 0.4 * differentiation)
(Note that I didn't bother to reduce the numbers in the total column 
to the appropriate number of significant digits.)

The data sources I used for this table are on pages 4, 14, and 43 of 
the NCDNHC report.

Note that my results match the ones on page 27 of the report - the  
remaining differences may be due to rounding errors in the weighting 
factors.



When I asked Alexander Svensson to independently verify my concerns, 
he came up with another problem: The table on page 14 
(responsiveness and governance rankings) is inconsistent in itself. 
GNR's score should be 27.75 (instead of 26.75, thereby placing GNR 
on rank 1, ahead of unity with 27.25), while ISOC's score should be 
23.25 (instead of 21.75; no ranking changes caused).


This error also sheds a spotlight on a methdological problem in the  
final evaluation of the NCDNHC's results: By merely averaging  
ranks, small differences in the originating scores (possibly caused  
by minor errors - the mistake in GNR's score corresponds to an error 
of about 3.5%!) are exaggerated in the end result.  In this  
particular case, for instance, the corrected "responsiveness" rating 
would place GNR on the same rank as Neustar in the average ranking 
evaluation on page 26 of the report.



I'll leave it to the NCDNHC team, ICANN staff, and the applicants  
themselves to check and verify the rest of the material provided.
-- 
Thomas Roessler                        <roessler at does-not-exist.org>