Article: "New judging system not necessarily a done deal"

Article: "New judging system not necessarily a done deal"

Post by Ann » Sat, 18 Jan 2003 22:59:04


I guess we could consider this the rebuttal to Bev Smith's Globe and
Mail article, although this time the people giving the opinions are from
the USFSA:

http://slam.canoe.ca/Slam030116/ska_new-ap.html

Although I have reservations, I also would like to keep an open mind
about the proposed new "points system", since I don't think it's
without some merit. I'm glad they, the ISU, seem to be seeking some
input from the skaters.

As long as they do away with the "anonymous judges and scoring" business
...

Ann

 
 
 

Article: "New judging system not necessarily a done deal"

Post by Ann » Sun, 19 Jan 2003 05:15:15

Quote:

> I guess we could consider this the rebuttal to Bev Smith's Globe and
> Mail article, although this time the people giving the opinions are from
> the USFSA:

> http://slam.canoe.ca/Slam030116/ska_new-ap.html

> Although I have reservations, I also would like to keep an open mind
> about the proposed new "points system", since I don't think it's without
> some merit. I'm glad they, the ISU, seem to be seeking some input from
> the skaters.

> As long as they do away with the "anonymous judges and scoring" business
> ....

> Ann

Additionally, Bev Smith herself reports a different perspective on the
proposed new ISU judging system:
http://www.globeandmail.com/servlet/ArticleNews/sports/RTGAM/20030117...

 
 
 

Article: "New judging system not necessarily a done deal"

Post by Fiona McQuarri » Sun, 19 Jan 2003 06:41:35

: I guess we could consider this the rebuttal to Bev Smith's Globe and
: Mail article, although this time the people giving the opinions are from
: the USFSA:

: http://slam.canoe.ca/Slam030116/ska_new-ap.html

: Although I have reservations, I also would like to keep an open mind
: about the proposed new "points system", since I don't think it's
: without some merit. I'm glad they, the ISU, seem to be seeking some
: input from the skaters.

When/how/where is the ISU doing this? So far they mostly seem to be
ignoring them, or sending out memos (e.g. $peedy re Yagudin) saying that
skaters who don't like the system don't understand them and that there has
to be a better job in getting the message out.

Fiona

 
 
 

Article: "New judging system not necessarily a done deal"

Post by BaleofA » Sun, 19 Jan 2003 07:43:16

Here's the part I found frightening:

<< An International Skating Union presentation at the U.S. championships
revealed yesterday some new details apparently adopted by the six ISU committee
members who are tweaking the system.

Everyone had already heard that 14 judges would be lined up to mark events
under the new system, but that a random system would select the marks of only
nine.

But with a trimmed mean, the two highest and two lowest marks would be tossed
out. That means that out of 14 original judges, only five marks would end up
counting. >>

From a statistical perspective, this seems questionable.  Anyone with a math
background wish to explain why (or how) this process would work?

LB

 
 
 

Article: "New judging system not necessarily a done deal"

Post by Fiona McQuarri » Sun, 19 Jan 2003 08:29:55

: Here's the part I found frightening:

: << An International Skating Union presentation at the U.S. championships
: revealed yesterday some new details apparently adopted by the six ISU committee
: members who are tweaking the system.

: Everyone had already heard that 14 judges would be lined up to mark events
: under the new system, but that a random system would select the marks of only
: nine.

: But with a trimmed mean, the two highest and two lowest marks would be tossed
: out. That means that out of 14 original judges, only five marks would end up
: counting. >>

: From a statistical perspective, this seems questionable.  Anyone with a math
: background wish to explain why (or how) this process would work?

It doesn't.
- According to the stats text in
my office (and this is just off the top of my head, people with better
statistical backgrounds feel free to correct me), in a population of 14, a
sample of 5 gives you at least a .0004 level of confidence that your
sample is representative - in other words, for every 1000 times you draw a
sample of 5 from that population of 14, 996 of those 1000 draws will truly
represent the opinions of the entire 14. This is statistically significant
(i.e. reliable) and would be considered meaningful in most situations.
HOWEVER this os only so if  there is a normal distribution among the
entire sample, i.e. that a certain percentage of the population falls
within a
certain distance of the mean (the % vary slightly depending on the size
of the population, but the general "shape" of the distribution is the
same, like a snake that swallowed a basketball).  
It's impossible to know whether
this is the case or not without seeing the entire set of 14 marks.
PLus...would you want to lose, say, Worlds if you happened to be the one
who got one of those 4 random draws that DIDN'T represent the opinions of
the entire panel?
 - Tossing out the high
and low
marks makes minimal difference to the mean.
- There is no guarantee that the mean or the marks the mean is
derived from truly represent the
performance being judged. In other words, if you have a panel full of
Balkovs, there will be a mean, but it might have nothing to do with the
quality of the performance. All the mean says is "here is a number that is
the average of all the marks awarded" - there is no compensation for the
fact that those marks might be flawed in some way.

Fiona

 
 
 

Article: "New judging system not necessarily a done deal"

Post by Altheal » Sun, 19 Jan 2003 14:18:40

BaleofAKS:
Quote:
>Everyone had already heard that 14 judges would be lined up to mark events

under the new system, but that a random system would select the marks of only
nine.  But with a trimmed mean, the two highest and two lowest marks would be
tossed out. That means that out of 14 original judges, only five marks would
end up counting.  From a statistical perspective, this seems questionable.
Anyone with a math background wish to explain why (or how) this process would
work?>

Poorly.

What matters is whether people are cheating.  Have one cheater among five is a
lot worse than having one cheater among four***.  And, a general rule of thumb
in stats is that the more information the better.  You don't just throw away
marks without reason...a smaller sample is more likely to be
non-representative.  There's just no reason to randomly select judges whose
marks will count -- now or in the new system.  (Obviously, I don't believe it
will make pressure or cheating less likely -- a country could still pressure
all 14 judges, JUST IN CASE each one's marks count.  There's nothing to lose
from doing so.  Particularly given that the judges are now freer to cheat,
because they are anonymous).

What might be interesting (and I gather this is what the USFSA is interested in
-- assuming they would benefit) is to talk about trying to figure out a way to
make judging panels more 'representative' of the entire pool of countries --
although I don't know if I believe that most panels aren't pretty
representative, and I'm not sure how one goes about making them more
representative.

-- Kate

 
 
 

Article: "New judging system not necessarily a done deal"

Post by Ann » Sun, 19 Jan 2003 21:41:44

Quote:


> : I guess we could consider this the rebuttal to Bev Smith's Globe and
> : Mail article, although this time the people giving the opinions are from
> : the USFSA:

> : http://slam.canoe.ca/Slam030116/ska_new-ap.html

> : Although I have reservations, I also would like to keep an open mind
> : about the proposed new "points system", since I don't think it's
> : without some merit. I'm glad they, the ISU, seem to be seeking some
> : input from the skaters.

> When/how/where is the ISU doing this? So far they mostly seem to be
> ignoring them, or sending out memos (e.g. $peedy re Yagudin) saying that
> skaters who don't like the system don't understand them and that there has
> to be a better job in getting the message out.

> Fiona

Toward the bottom of the article it mentions that Paul Wylie is one of
the people providing input into the "proposed system".  Perhaps I should
have said "former skaters".  Was Yagudin's beef with the "proposed
system" or the "interim system"?

Ann W.

 
 
 

Article: "New judging system not necessarily a done deal"

Post by cmh.. » Tue, 21 Jan 2003 22:35:30



Quote:
>It doesn't.
>- According to the stats text in
>my office (and this is just off the top of my head, people with better
>statistical backgrounds feel free to correct me), in a population of 14,
>a  sample of 5 gives you at least a .0004 level of confidence that your
>sample is representative - in other words, for every 1000 times you draw
>a  sample of 5 from that population of 14, 996 of those 1000 draws will
>truly  represent the opinions of the entire 14. This is statistically
>significant  (i.e. reliable) and would be considered meaningful in most
>situations.  HOWEVER this os only so if  there is a normal distribution
>among the  entire sample, i.e. that a certain percentage of the
>population falls  within a
>certain distance of the mean (the % vary slightly depending on the size
>of the population, but the general "shape" of the distribution is the
>same, like a snake that swallowed a basketball).  

snip

Fiona:

I think you are not applying your statistics properly. I believe you are
taking 5/14 and applying that to sampling from a large population. In the
judging case, the total population is small and the sample "n" is
extremely small. The central limit theorem does not apply and you have to
look at binomial distributions (like calculating the odds at craps). The
exact odds of getting a right or wrong sense of the total poulation will
depend on the nature of the question you pose, but the odds of getting an
unrepresentative sample will be much greater than what you estimated. Just
look at it this way: a selection of any sample of 5 should have an average
standard deviation of 5/sqrt(5), which is a 44% error!

Chris

--
Chris M. Hall, Assistant Research Scientist
Dept. of Geological Sciences, University of Michigan

Quando omni flunkus moritati

 
 
 

Article: "New judging system not necessarily a done deal"

Post by Fiona McQuarri » Wed, 22 Jan 2003 05:08:51



:>It doesn't.
:>- According to the stats text in
:>my office (and this is just off the top of my head, people with better
:>statistical backgrounds feel free to correct me), in a population of 14,
:>a  sample of 5 gives you at least a .0004 level of confidence that your
:>sample is representative - in other words, for every 1000 times you draw
:>a  sample of 5 from that population of 14, 996 of those 1000 draws will
:>truly  represent the opinions of the entire 14. This is statistically
:>significant  (i.e. reliable) and would be considered meaningful in most
:>situations.  HOWEVER this os only so if  there is a normal distribution
:>among the  entire sample, i.e. that a certain percentage of the
:>population falls  within a
:>certain distance of the mean (the % vary slightly depending on the size
:>of the population, but the general "shape" of the distribution is the
:>same, like a snake that swallowed a basketball).  

: snip

: Fiona:

: I think you are not applying your statistics properly. I believe you are
: taking 5/14 and applying that to sampling from a large population. In the
: judging case, the total population is small and the sample "n" is
: extremely small. The central limit theorem does not apply and you have to
: look at binomial distributions (like calculating the odds at craps). The
: exact odds of getting a right or wrong sense of the total poulation will
: depend on the nature of the question you pose, but the odds of getting an
: unrepresentative sample will be much greater than what you estimated. Just
: look at it this way: a selection of any sample of 5 should have an average
: standard deviation of 5/sqrt(5), which is a 44% error!

Hi Chris,
Yes, you are correct, and thanks for the clarification.

fiona