Judging

Judging

Post by Virgin » Sat, 30 Mar 2002 15:57:46



Quote:
>I'd be curious to know her thoughts on secrecy about the markings.  I think it
>would be educational for the skaters while forcing the judges to be more honest
>if they gave a brief presentation on why certain skaters were marked as they
>were.  I would think this would be best after the judges meeting, of course,
>and it would be unwieldly to do it for each skater, but I do think this
>information should be available to all who want it.  There's probably a reason
>it's not realistic, but I'd be interested in hearing it.

A proposal: Record the judges meeting after the event. Have each judge
announce justifications for say the top x placements(like you say it
would be unwieldy to do it for all) plus any justifications the
referee(s) may want the judges to provide. Also, discuss any other
important issues that might have presented themselves during the
competition. After the judges meeting, someone could take the meeting
record and create an overview document, make copies, and have it
available to the public(reporters and other interested parties
including skaters, coaches, etc) the morning after the event. Not
every detail has to be in this document, but have enough so that
people know where the judges were coming from.

-----------== Posted via Newsfeeds.Com, Uncensored Usenet News ==----------
   http://www.newsfeeds.com       The Largest Usenet Servers in the World!
------== Over 100,000 Newsgroups - Ulimited downloads - 19 servers ==-----

 
 
 

Judging

Post by Jonafa » Sat, 30 Mar 2002 20:52:02

Hi, Janet,

Good post.  Lots to chew on.

My daughter has a question about synch skating judging that perhaps you
would comment on.

She is a member of a champion Jr. team and has done quite a bit of
travelling this year.

She thinks that in synch skating, each judge ought to be responsible for a
certain aspect of the program:  footwork and edges, grips and upper body,
stroking and power, maneuvers, musicality and presentation, etc.  She thinks
it's too easy for judges to miss stuff in synch (bad stuff and good stuff).

I'm pretty certain they do this is ISI synch, but no one seems to think it
is a good thing.

I have to admit I think there is merit in the idea for synch (not for
singles/pairs/dance).  I've seen stuff missed by the parents, who when they
watch the video later say, "Wow, I never even saw that."  (Yes, I know that
parents are not judges!)

Any thoughts?



Quote:
> (I haven't been in hiding, I just haven't been able to keep up.   I set my
> newsreader to display messages only from the past 14 days, so if 2 weeks
go by
> and I haven't gotten to something, I don't see it.    If you've asked me a
> question and I haven't answered it ....... that's why)

> ANYWAY ..... I know that there have been lots of messages about judging,
and
> discussion about various proposals.   This message isn't in response to
any one
> of them.   It's approximately a copy of a message I posted on Skatefans.

> Here are my modest proposals for addressing some judging problems.   I
have
> chosen these particular avenues  because I believe that these could
actually be
> accomplished, and might be passed by the ISU, and because they do not have
an
> impact on the skaters themselves (except to the extent that skaters may be
less
> subject to incorrect judging).

> 1.   Admit that some things such as "cultural preferences and background"
> are probably impossible to eradicate, and thus make a focus minimizing its
> impact on the skaters.  (more about this below)

> 2.   Do statistical studies along the lines that Dirk Schaeffer has done.
> Publish the results.    If considered necessary for publication, remove
names
> and substitute country designations.  If the results are similar to
Dirk's,
> [who finds that gross national bias is not rampant] use the opportunity of
> publication to point out that statistical studies do not support the
notion of
> widespread egregious national bias on the part of judges.    If one or a
few
> judges are "uncovered" in this process (for instance, Dirk's studies DID
> identify a small number of judges whose judging tendencies fell outside
the
> statistical norms in terms of favoring their own skaters.   At least one
of
> these judges was disciplined by the ISU for national bias -- not because
of
> Dirk's study, I hasten to add)), DEAL WITH IT (see below).

> 3.   "Routinize" this kind of statistical study, so that the judges, the
> skaters, and the public understand that the ISU is taking this step to
> identify potential problems on an ongoing basis.

> 4.   Explain to the judges (and everybody else) that the object should
> always be to come up with the most nearly correct placements, and that all
> efforts are aimed in that direction.   Explicitly state that things like
> trading
> favors, exerting or yielding to pressure, "looking the other way", or
> failing to recognize in onesself a tendency not to recognize or
acknowledge
> virtues in skating that is not "your preferred type" from a cultural
prerence
> point of view run counter to the primary objective of judging, and they
are ARE
> NOT, and SHOULD NOT BE regarded as "a norm" in judging, no matter what may
be
> true of a society at large

> 5.  Develop some examples, training films, a curriculum, and workshops to
> address the issue of identifiying and recognizing "cultural preferences"
in
> weighting qualities of skating.   Discuss what things are valued highly by
> certain groups, and why.   Discuss and instruct about which items have a
> legitimate higher value than others, and about treading a middle ground
and
> not allowing great skill in X to discount poor execution in Y.

> 6.   Address BOTH the reality and perception of unfairness through
possible
> national bias or cultural preferences by devising rules to assure a broad
> geographic representation on all international panels.   (e.g.  divide the
> world into "Europeans countries" and "4Cs countries" and state that no
> international panel may have more than a bare majority from either group.
Or
> divide the world into three regions (such as Europe; Asia; The
Americas/Oceania
> and state that panels must be divided 2/2/3 or 3/3/3).    Where one region
gets
> more judges than another, rotate between/among regions as to who gets the
> greater number next time.   For instance, if the 4Cs countries got 5
judges on
> the Worlds Dance panel this year, they get 4 next year.

> 7.   Address the possible reality and the perception of deal-making among
an
> in-crowd by limiting the number and array of events that any particular
> judge may judge in a given year.   This may mean that some countries are
> (relatively speaking) under-represented if they have very few qualified
> judges.   On the other hand, it might actually INCREASE overall
> representation, as more countries in total might be represented
> on international panels over the course of a year.    I believe that
having the
> same judge on the same panel for multiple events with the same skaters in
the
> same year (as in, judging men at a GP event, Worlds, and Olympics) makes
for
> worse judging, in that it may lead a judge to develop (inadvertently but
> inevitably) expectations that would interfere with the fairest possible
> judging.    I also believe that having the same judges at multiple events
> increases the ease with which deals may be made, through having it be
partly
> the "same cast of characters" from event to event.   It also reinforces a
kind
> of "in crowd" caste system, AND it reduces the ability to get more
valuable
> experience to more judges.

> 8.   Revise the rules and punishments for improper conduct so as to (1)
> encourage the reporting of pressure and deals (currently, it would seem
that
> the consequences to the judge of reporting a problem are such that they
are
> reluctant to do so), and (2) increase the seriousness of the consequences
to
> a judge or official or NGB found to be guilty of impropriety:

>    -- a judge who reports pressure being exerted on her/him to mark in a
> certain way should -- depending on the circumstances (including how far in
> advance it is reported, the nature and extent of the pressure), MIGHT
remain on
> the panel, but in most cases should be removed from that panel WITHOUT
> PREJUDICE, and should be placed on another panel as soon as possible.
Having
> the normal reaction be removal from the panel would decrease the
temptation to
> falsely report pressure as a means of "adjusting" the makeup of a panel.
If
> the person accused of exerting pressure is another judge on the panel,
that
> person should also be removed WITHOUT PREJUDICE and replaced by another
judge
> from her/his "region" by random draw among "regional" NGBs.     If the
accused
> is actually found to have attempted to exert improper pressure, then the
> punishments below should be exacted.   If the accusation is determined to
have
> been unfounded, that judge should be placed on a future panel as soon as
can be
> managed.

>    -- a justification for removing judges without prejudice is that a
judge who
> has been the subject of attempts to influence her/him to exert pressure
may
> find it difficult (no matter how hard they try) to render a completely
> impartial judgement.

>     -- a judge who is found, through statistical studies to demonstrate a
> pattern of greater-than-average national bias should be restricted from
> international panels for a year.   The judge should complete a prescribed
> "course of study" about bias.   The next year, the judge should be given
the
> option of "trialing" at international competitions, and having her/his
results
> monitored as if s/he were still an "apprentice".   The first real
assignments
> after suspension ..... for a period of perhaps 3 years, should be at minor
> international competitions, and subject to greater-than-normal scrutiny.

>      -- for judges suspected of having yielded to pressure, or to have
> traded favors, or to have colluded with others, or to have maliciously
made
> false accusations in an attempt to remove a particular judge or NGB from a
> panel,  there should be an immediate suspension "without prejudice" until
the
> matter can be investigated and conclusions drawn.   The loss of that judge
from
> an upcoming panel already assigned should be filled by random draw from
the
> judge's "region", rather than by replacement from the NGB.

>      -- for judges found to have yielded to pressure, or to have traded
> favors, or to have colluded with others, or to have made false accusations
> maliciously in an attempt to alter the makeup of a panel,  there should be
a
> MINIMUM of an immediate and non-negotiable four year suspension, and a
MAXIMUM
> of permanent loss of accreditation.  For those who have been suspended, a
> return to judging should be first at minor internationals, and the judge's
> activities should be subject to close scrutiny for a period of not less
than
> five years following return to judging.

>      -- Officials other than judges should be subject to analogous
> punishments.   this includes members of ISU committees, and officials of
NGBs
> who may or may not be serving as an official in the particular
competition,

>      -- If a national governing body is found, through any of its
officials, to
> have engaged in activities that encouraged, condoned, instigated, or
performed
> trading of favors, collusion,

...

read more »

 
 
 

Judging

Post by Trudi Marrapo » Sat, 30 Mar 2002 20:12:31

I really like janet's proposals, although I share Fiona's concerns, too. I
only wish someone from the ISU who had any sort of "pull" was reading and
absorbing all these ideas. (Maybe someone from the USFSA?)

I had a thought to add another penalty to the NGB proved to be cheating:
that all its skaters be pulled from competition for one year.

I know, I know, I know...I can hear the screaming already. "What a
terrible idea! The SKATERS didn't cheat. It's UNFAIR to make them suffer
for the sins of their NGB."

Yes. It is. Terribly unfair.

But it might be the only way to get the damn NGBs to shape up or ship out.
If they knew their hanky-panky might not just get THEM in trouble, but
have the exact OPPOSITE effect of what they wanted...i.e., result in their
skaters not even being permitted to compete at all...it just might be a
stronger deterrent.

I'm sure many people think this would be a horrible idea. I'm not
necessarily saying it would be a great idea myself. But I'm wondering.
After all, when a college in the NCAA system is found guilty of a
sufficient number of recruiting violations, the team is barred from the
playoffs. And that's not really fair to all the athletes
either...especially the ones who are on the team but did not do anything
wrong...but it happens.

I don't know. Maybe someone who knows more about the NCAA will claim that
such suspensions don't do jack to clean up the system or serve as a
deterrent. I welcome any and all comments.
--
Trudi
adding the question "What do you mean?" to every post, just in case...

 
 
 

Judging

Post by Carla Jenki » Sat, 30 Mar 2002 22:02:31

<A proposal: Record the judges meeting after the event. Have each judge
announce justifications for say the top x placements(like you say it
would be unwieldy to do it for all) plus any justifications the
referee(s) may want the judges to provide. Also, discuss any other
important issues that might have presented themselves during the
competition. After the judges meeting, someone could take the meeting
record and create an overview document, make copies, and have it
available to the public(reporters and other interested parties including
skaters, coaches, etc) the morning after the event. Not every detail has
to be in this document, but have enough so that people know where the
judges were coming from.>***ia

   That is an excellent idea.
    In high school Declamation, our judges went to this much effort.  We
received our judges" sheets a week following the competition.  By then,
we students were able to rationally evaluate constructive criticism.    
    I, too, do not think that the present system is the problem.
Although I have referred to myself as Judge #14, I do not seriously
think that there should be that many judges and seven of these scores
randomly picked.  Secrecy will not solve the present dilemma--and it may
make things worse.  
    I like the idea of a head judge or referee being able to call an
immediate conference if a score or placement was .5 higher or lower than
the norm.  
   Carla

http://SportToday.org/

 
 
 

Judging

Post by Locutus of Bo » Sun, 31 Mar 2002 00:41:52

Quote:
>After all, when a college in the NCAA system is found guilty of a
>sufficient number of recruiting violations, the team is barred from the
>playoffs. And that's not really fair to all the athletes
>either...especially the ones who are on the team but did not do anything
>wrong...but it happens.

That's not the proper analogy. The proper one is the "death penalty", which was
levied against my alma matter because of alumni violations in the 1980's. SMU's
football program was killed. When it was allowed to return, it has had a
horrible time recruiting any players of any skill, and after having been the
most-winning team in college football in the late seventies, it now can't even
muster a winning season.

Quote:
>I don't know. Maybe someone who knows more about the NCAA will claim that
>such suspensions don't do jack to clean up the system or serve as a
>deterrent. I welcome any and all comments.

The "death penalty" did absolutely nothing to clean up the system nationwide,
and made a significant economic dent in the university-area cities and  caused
the university to depend more on its generous alumni contributions
(sustentation drives) rather than athletic department revenues for a period of
several years.

Now... I don't think we can compare a private not-for-profit NGB and its
skaters in a relatively unpopular discipline in the USA to anything the NCAA
does. I'm more curious how suspending dance from the OOOOO will affect things
in countries where ice dance is actually popular... and that's not the USA.

Peg
==
rec.sport.skating.ice.figure FAQ (Frequently Asked Questions):
   http://www.frogsonice.com/skateweb/reference.shtml
Newsgroup Intro and Netiquette:
   http://www.frogsonice.com/skateweb/faq/introduction.shtml

 
 
 

Judging

Post by mmarte » Sun, 31 Mar 2002 01:02:03

Random musings on the discussion of statistical studies:

1. Although I realize it's possible to figure out systematic bias of
individual judges and biases of "blocs" I think its still a bit
problematic to infer bias for or against particular skaters because of
the small "n" problem--we aren't talking about groups of 100 skaters
here, usually we are only talking about the placement of the top 10 or
smaller still, the podium at a series of events.

2.Is there a systematic bias toward all the skaters in the top 10 that
puts them there and makes it impossible for the bottom 20 to get to
the top?  Is the skater from Mexico in the bottom 20, 20% because of
their lack of skills, 20% because of simple expectations (their
placements at previous comps) and 60% because they have no favorable
judge?

3. A simple averaging of the scores also doesn't capture strategic
ranking that goes on to get a particular skater a particular placement
and it doesn't capture the contingency factor--skaters who do so
obviously well or so obviously badly that the judges feel some
pressure to take the actual performance into account.  This doesn't
seem to matter as much to the ice dance judges but I think it has been
rumored/reported that Sarah's gold was partly a result of a***up
in a bloc deal--they wanted Irina to beat Michele but because they
didn't succeed in getting it together in the SP *and Michele skated
very well* they couldn't make Irina the overall winner, which left the
door open to Sarah.

4. The 14 judge system is likely at least in the short term, IMHO, to
intensify national bias.  If you are a Yuri Balkov, you now don't know
who will place your skaters highly or whose scores will count so you'd
damn well better place them as high as possible yourself, just in
case. The marks will be all over the map and the winner will be a
surprise, created out of the average of scores, minus the eliminated
randoms.  Over time perhaps after many iterations and with sufficient
scrutiny of judges' marks to penalize consistent national bias maybe
the system would work the way it is intended.

5. I think investigating judges actions/remarks is a much more
promising way to go.  Also publishing comments after an event along
with the marks. The ABC interview with selected judges from the pairs
SLC at Worlds gave some insight into their thinking processes and it
would be good to see more of that. It would make it harder for a
biased judge to overrate a clearly inferior skater or performance
without looking corrupt or foolish. As was discussed on another
thread, presenting the base marks for the technical program with the
marks given would force judges to be honest about discerning mistakes
from more basic criticisms like, "I didn't like the program," "the
lifts were ugly," "it was an old program."

mm

 
 
 

Judging

Post by janet swan hi » Sun, 31 Mar 2002 04:08:37

Fiona says:

Quote:
>I agree with most of the sensible ideas, BUT....doing a statistical
>analysis like Dirk's (and I believe some of us have tried to debate this
>point with Dirk) is based on the assumption that the norm of a set of
>marks is valid.
 (snip)
>I honestly don't know how, at least numerically, you distinguish between
>an "outlier" who is marking honestly and an "outlier" who is a corrupt
>marker.

And what she says makes sense.   You may note that the only use I suggested
putting the statistical studies to was to try to detect a level of
nationalistic marking that is statistically above the norm.    That is, limited
to one special kind of "outlier"  (they already probably keep track of other
kinds/degrees of deviation from the determination of the panel anyway).    

And of course, such studies would have to be longitudinal ....... you can't
draw any sort of a valid statistical conclusion from a single dataset.      
janet

 
 
 

Judging

Post by janet swan hi » Sun, 31 Mar 2002 04:11:21

Quote:

>I had a thought to add another penalty to the NGB proved to be cheating:
>that all its skaters be pulled from competition for one year.

My idea was to pull the that NGB's judges, referees, accountants, etc.  
Personally, I think that that might have a greater impact on the people in a
position to make changes than pulling the skaters would.

janet

 
 
 

Judging

Post by janet swan hi » Sun, 31 Mar 2002 04:32:22

Daria asks:

Quote:
>I'd be curious to know her thoughts on secrecy about the markings.  I think
>it
>would be educational for the skaters while forcing the judges to be more
>honest
>if they gave a brief presentation on why certain skaters were marked as they
>were.  I would think this would be best after the judges meeting, of course,
>and it would be unwieldly to do it for each skater, but I do think this
>information should be available to all who want it.  There's probably a
>reason
>it's not realistic, but I'd be interested in hearing it.

I believe that much value could be derived from greater openness about judges
markings.  I feel confident that a practice could be designed that enabled
judges to explain why they had ranked or marked skaters as they had, that was
not confrontational, but allowed for some two-way communication.    I believe
that the degree of secrecy/confidentiality that is current practiced is not in
the overall best interest of the sport.   Providing and outlet for orderly
communication with the public is NOT equivalent to approving a free-for-all.  

When I judge the little local competitions that I judge, I often wish that
there were a mechanism (other than having individual coaches take individual
initiative and ask) for communicating some salient points about a particular
event.

For instance, in events where the registration information says that "only two
double jumps in total may be attempted, chosen from the list of double loop,
toeloop, and salchow",  judges take deductions from skaters who violate the
rules (for instance, if a skater does a 2L, a 2T and a 2S/L).   Other skaters
and coaches are insensed when they see this, and are beside themselves if that
skater wins the event or places well.    I would love to be able to say that I
had taken a 0.2 deduction for the extra/added element, (and that ( set a
basemark for what the program would have been WITHOUT that element) but that
the number, nature, and quality of other elements was sufficient to justify a
mark that more than made up for the deduction.  

I imagine that judges who are dealing with much higher stakes events are no
less desirous of an opportunity to explain themselves.  

It hasn't been a particularly comfortable time to be a judge at ANY level in
the last few months, as the distant actions of a few have cast doubt on the
integrity of us all.     The judging ITSELF remains intensely satisfying, as
the ability to give something to the sport and the people in it continues, but
the realization that there are many people out there who believe that judges
are capricious and venial doesn't exactly put joy in my heart.     I'd like to
have better opportunities to help people understand why I've done what I've
done, so that they can improve their skating, their knowledge, and their
enjoyment.  

janet

 
 
 

Judging

Post by Pat » Sun, 31 Mar 2002 07:11:46


(snip)

 but the realization that there are many people out there who believe
that judges are capricious and venial doesn't exactly put joy in my
heart.    

No, it won't until the judging is perceived to be honest and fair at
an international level.  What runs downhill?  (wry smile)  You're (not
you personally but all lower level judges) are close and handy.  

I'd like to

Quote:
>have better opportunities to help people understand why I've done what I've
>done, so that they can improve their skating, their knowledge, and their
>enjoyment.  

So do a lot of international level judges.  So many like you, want the
opportunity to say why they've marked the way they have based on the
rules.  They love the sport, admire the skaters who do the sport, and
would like to feel of value rather than a detriment.  (sigh)

I just wish the ISU would *make it so*.

One thing that has never been mentioned, and I'll do it now (apologies
if it has) is that figure skating and speed skating have to be
separated.  Cinquanta may be ok in speed skating, he's just not in
figure skating.  And why should he be?  He's a speed skater.  It is
long past the time when the two should be separated.  I think that
wouldn't hurt.  

Pat C

http://www.skatemusiclist.com

 
 
 

Judging

Post by Trish O'Bri » Sun, 31 Mar 2002 10:38:01


Quote:

> >I had a thought to add another penalty to the NGB proved to be cheating:
> >that all its skaters be pulled from competition for one year.

> My idea was to pull the that NGB's judges, referees, accountants, etc.  
> Personally, I think that that might have a greater impact on the people in a
> position to make changes than pulling the skaters would.

> janet

I see your point Janet, but wasn't the the basis of the alleged
deal-making in SLC based on the fact that France didn't have a judge
on the dance panel? I'd be concerned that removing the
"behind-the-scenes" folks you mentioned would foster *more*
deal-making, not less.

I also think that whatever sanction is applied should run through the
Olympic cycle, as this tends to be the driver of a lot of the rumored
deals - getting skaters or teams well positioned for the OOOOOs.

Trish, posting from Google, since attbi sucks....

 
 
 

Judging

Post by Trudi Marrapo » Sun, 31 Mar 2002 20:36:50



Quote:
> >After all, when a college in the NCAA system is found guilty of a
> >sufficient number of recruiting violations, the team is barred from the
> >playoffs. And that's not really fair to all the athletes
> >either...especially the ones who are on the team but did not do anything
> >wrong...but it happens.

> That's not the proper analogy. The proper one is the "death penalty",
which was
> levied against my alma matter because of alumni violations in the
1980's. SMU's
> football program was killed. When it was allowed to return, it has had a
> horrible time recruiting any players of any skill, and after having been the
> most-winning team in college football in the late seventies, it now can't even
> muster a winning season.

I do remember when this happened.

Quote:
> >I don't know. Maybe someone who knows more about the NCAA will claim that
> >such suspensions don't do jack to clean up the system or serve as a
> >deterrent. I welcome any and all comments.

> The "death penalty" did absolutely nothing to clean up the system nationwide,
> and made a significant economic dent in the university-area cities and  caused
> the university to depend more on its generous alumni contributions
> (sustentation drives) rather than athletic department revenues for a period of
> several years.

> Now... I don't think we can compare a private not-for-profit NGB and its
> skaters in a relatively unpopular discipline in the USA to anything the NCAA
> does. I'm more curious how suspending dance from the OOOOO will affect things
> in countries where ice dance is actually popular... and that's not the USA.

> Peg

Well...you say it did SMU tons of damage, and hurt a lot of peripheral
people...I'm wondering, if an NGB knew that it might destroy its skating
program by engaging in cheating, would it be so quick to do so?

Right now, I'm thinking of Austria, actually. Look in any history of
skating and you can see that in the early days, the Austrians used to win
medal after medal. You can also find many indications that the Austrian
NGB were the champions at vote-fixing and influence peddling. Well...where
is the Austrian program now? In the pits.
--
Trudi
adding the question "What do you mean?" to every post, just in case...

 
 
 

Judging

Post by soobell » Mon, 01 Apr 2002 06:36:08


| One thing that has never been mentioned, and I'll do it now
(apologies
| if it has) is that figure skating and speed skating have to be
| separated.  Cinquanta may be ok in speed skating, he's just not in
| figure skating.  And why should he be?  He's a speed skater.  It is
| long past the time when the two should be separated.  I think that
| wouldn't hurt.
==
And if they wish to keep both under one banner organization at the
very top, to share revenues, they can otherwise separate it with a VP
of speed skating and a VP of figure skating.  Then we can have two
VIP's for skating instead of one - double your pleasure, double your
fun <g>.

soobelle

 
 
 

Judging

Post by StackDum » Fri, 05 Apr 2002 08:27:29

Quote:

>> information should be available to all who want it.  There's probably a
>reason
>> it's not realistic, but I'd be interested in hearing it.

>It would embarrass the***ens out of the judges. If
>a test could be done so that each judge had to comment
>without the other judges hearing those comments, you
>would find quickly that the explanations were bizarre
>and only vaguely related to each other.  It should be
>done while the viewer was viewing the skaters
>program in secret from the judges comments. Very
>informative too. An even more informative test would
>be to have each judge look at a recorded program,
>independantly, and explain what they saw !!!!!  as
>they took each deduction. Have them do it twice
>on 2 seperate days over a span of say 10 recorded
>programs. Then you'll find out the true meaning of
>random.

>johns

I'd  LOVE to see this!   But it'd only happen in maybe a Jersy
Kazinski novel.  LOL

- Show quoted text -