| |
Nightlord
Account closed
Registered: Jan 2003
Posts: 131 |
weighted average?
ok something i just noticed when i was voting for alih.
the guy has 6 9's and 3 8's and a 1 from a downvoter. his weighted average turns out to be 7.7.
now if we took the arithmetic average. it would be
(6 x 9) + (3 x 8) + 1 = 79 divided by 10 would be ~ 7.9
so the weighted average function was supposed to be able to recognize downvoting and perform better than arithmetic average. I say something is wrong.
not only the real average should have been somewhere in high 8's, but also the csdb function works even worse than the arithmetic average.
am i making a calculation mistake or something? |
|
| |
Perff
Administrator
Posts: 1679 |
I can see how that looks strange.
I just took a look at the algorithm that calculates the rating, and I found a little thing that might be an improvement.
Now the rating is 8.5, which should be a lot better than 7.7. :)
It's still not the perfect 8.6666666666... as it should be if you removed the 1, and made a regular average, but at least it's close. More (real) votes will bring it even closer.
I recalculated the rating for all entries in CSDb, and my little alteration had no effect on the top entries in the charts. It only had an effect on entries with few votes where there was some very extreme votes, like in this case. |
| |
WVL
Registered: Mar 2002
Posts: 902 |
Quote: I can see how that looks strange.
I just took a look at the algorithm that calculates the rating, and I found a little thing that might be an improvement.
Now the rating is 8.5, which should be a lot better than 7.7. :)
It's still not the perfect 8.6666666666... as it should be if you removed the 1, and made a regular average, but at least it's close. More (real) votes will bring it even closer.
I recalculated the rating for all entries in CSDb, and my little alteration had no effect on the top entries in the charts. It only had an effect on entries with few votes where there was some very extreme votes, like in this case.
no effect? :) Perff finally managed to kick Crossbow and Graham out of the top 3 coders :DD |
| |
Perff
Administrator
Posts: 1679 |
Hmmm... oups! :) Missed that. hmm.. checking.. |
| |
WVL
Registered: Mar 2002
Posts: 902 |
Quote: Hmmm... oups! :) Missed that. hmm.. checking..
why check that? :) I like it ;) now they have to come up with another cool demo to gain some points :) |
| |
Perff
Administrator
Posts: 1679 |
Just checking if my alteration made people with only bad votes get a rating of 15 or something. ;P
But except for the two people with very few votes (always hard to do a good calculation on few data), I can't see that the rating calculated is unfair, so Graham & Crossbow better shape up! ;);) |
| |
Hein
Registered: Apr 2004
Posts: 954 |
It's still a bit odd to see a MacGyver in the Cracker charts, though he is a pr manager (with 1.7 as vote).. |
| |
Jazzcat
Registered: Feb 2002
Posts: 1044 |
I guess the charts on CSDB can only be taken as serious as those who lodged the votes to begin with. ;)
Charts these days are a laugh, a real shame some old-style scene chart/website is not compiled/created. It may stir some competition or provoke some motivation. |
| |
Nightlord
Account closed
Registered: Jan 2003
Posts: 131 |
there needs to be a present chart like it is supposed to be on diskmags. a chart where someone who has not released anything for a certain time should be left out of.
|
| |
Celtic
Administrator
Registered: Jan 2002
Posts: 807 |
i think instead of 5 votes a production, person or group should get 10 votes to be entered in the charts. there are enough users nowadays, and someone with only 5 votes usually ends up at a place where they dont belong. |
| |
Perff
Administrator
Posts: 1679 |
Quote: i think instead of 5 votes a production, person or group should get 10 votes to be entered in the charts. there are enough users nowadays, and someone with only 5 votes usually ends up at a place where they dont belong.
I though about something similar. This would make the charts a bit more realistic, but on the other hand a lot of entries will be removed from the charts. (The ones with 5-9 votes)
Anyone against the idea? |
| |
Nightlord
Account closed
Registered: Jan 2003
Posts: 131 |
10 seems to be a good number.
perff. do you ever plan to make a chart where inactive people are dropped after a certain time? |
| |
Perff
Administrator
Posts: 1679 |
Quoting nightlordperff. do you ever plan to make a chart where inactive people are dropped after a certain time?
No, not really. I always though of the CSDb charts as a kind of all-time-chart. |
| |
Hein
Registered: Apr 2004
Posts: 954 |
It would be nice to add something like number of voters in the calculation.. Popularity is also important, the more votes, the popular the guy/girl.. |
| |
Perff
Administrator
Posts: 1679 |
Note taken. Actually there is some kind of this already. :)
If you take a look at the ratings, then those with a high number of votes match the arethmetic average of the votes (perhaps with some extreme fake votes removed first). If you then take a look at the entries with only a few votes, you might notice that their rating is less than the arethmetic average.
Now I'm talking in general terms. There might be a few entries where this is not completly accurate.. |
| |
Tch
Account closed
Registered: Sep 2004
Posts: 512 |
Quote: 10 seems to be a good number.
perff. do you ever plan to make a chart where inactive people are dropped after a certain time?
Hmm,voting doesn´t seem popular.
Must admit that I am not a dedicated voter either.
Raising the bar to 10 votes might be pushing it.
Seeing "Tsunami" only recieving 42 votes,I think a lot of lesser productions will just slip into forgottenness..
It would be a shame.
The barrier of 5 votes seems hard enough to break.
It would be quite ridicilous to see a scene that has been around for over 20 years,be represented by lists with only 50 releases.
The same goes (especially) for scener-lists.
Take a look and see who will be missing.
It´s like creating a history that never happened. |
| |
Zyron
Registered: Jan 2002
Posts: 2381 |
I second that. |
| |
Perff
Administrator
Posts: 1679 |
As someone might have noticed I have (once again) altered the weighted average algoritm a tiny bit, so for instance MacGyver dosn't get a 10 as cracker.
For now we will leave it at 5 votes as it have been so far.
If you still think some of the ratings looks weird or are unfair then remember the following:
1. The charts here in CSDb are only for the fun of it. The main purpose of CSDb is still information = facts.
2. The more votes, the more accurate the ratings are, so VOTE! and get others to vote too. :) |
| |
TDJ
Registered: Dec 2001
Posts: 1879 |
Quote: As someone might have noticed I have (once again) altered the weighted average algoritm a tiny bit, so for instance MacGyver dosn't get a 10 as cracker.
For now we will leave it at 5 votes as it have been so far.
If you still think some of the ratings looks weird or are unfair then remember the following:
1. The charts here in CSDb are only for the fun of it. The main purpose of CSDb is still information = facts.
2. The more votes, the more accurate the ratings are, so VOTE! and get others to vote too. :)
I'd vote more if I had the feeling my vote actually counts.
And yes, that's a reference to the transparent voting discussion ;) |
| |
Ben
Account closed
Registered: Feb 2003
Posts: 163 |
Perff, I reckon Bayesian weighing (i.e. using the empirical distribution a priori) will reflect the actual appreciation better than does a rather arbitrary appropriation of probablistic weighing. |
| |
Earthshaker
Registered: Sep 2002
Posts: 118 |
Quote: Perff, I reckon Bayesian weighing (i.e. using the empirical distribution a priori) will reflect the actual appreciation better than does a rather arbitrary appropriation of probablistic weighing.
Same goes for me! ;) |
| |
Bubis
Account closed
Registered: Oct 2012
Posts: 10 |
I've got a related question, I hope this is the right place to ask.
So, one of my entries got 2x10 and 3x9, how comes that my weighted average is 9.8? What is the formula? :) |
| |
chatGPZ
Registered: Dec 2001
Posts: 11384 |
about time this thread got necro-ed |
| |
Oswald
Registered: Apr 2002
Posts: 5094 |
Quote: I've got a related question, I hope this is the right place to ask.
So, one of my entries got 2x10 and 3x9, how comes that my weighted average is 9.8? What is the formula? :)
the formula is secret, to avoid manipulating votings. but it has some secret ingredient against downvoters. so likely alone standing votes, or those not so numerous has less weight |
| |
Jammer
Registered: Nov 2002
Posts: 1335 |
I applied my compo voting formula to musicians' charts for fun:
25Hz Compo average calculator
It's simple like hell but question is if it does the right job, besides weeding out unpopular votes:
(10 * #10s^2 + 9 * #9s^2 + 8 * #8s^2 + 7 * #7s^2 + 6 * #6s^2 + 5 * #5s^2 + 4 * #4s^2 + 3 * #3s^2 + 2 * #2s^2 + #1s^2) / (#10s^2 + #9s^2 + #8s^2 + #7s^2 + #6s^2 + #5s^2 + #4s^2 + #3s^2 + #2s^2 + #1s^2)
It surely inflates average if devote fans and groupmates throw their handful of 10s but this effect applies equally to every popular value so it's honest in this respect. |
| |
F7sus4
Registered: Apr 2013
Posts: 117 |
I remember seeing a video here at the forums where one guy did vote 1 to 10 to see how the voting system behaves. The conclusion was that it more-or-less ignores a few 1-3 point votes before starting to take them into the equation. Which is why downvoting with 4 became popular.
Note that having public votes would prevent most of it. |
| |
Krill
Registered: Apr 2002
Posts: 2979 |
How does it perform against CSDb Voter Pro 1.0 ? :) |
| |
Jammer
Registered: Nov 2002
Posts: 1335 |
Quoting F7sus4I remember seeing a video here at the forums where one guy did vote 1 to 10 to see how the voting system behaves. The conclusion was that it more-or-less ignores a few 1-3 point votes before starting to take them into the equation. Which is why downvoting with 4 became popular.
Formula is not weighted at all (but can be, like everything else). You make things bad with downright 1 but you need quite a lot of 1s so lonely vigilantes stand no bigger chance against cluster of other grades ;)
Quoting F7sus4Note that having public votes would prevent most of it.
I'd opt for public votes as well but in practice it prevents people mostly from voting at all :D
Quoting KrillHow does it perform against CSDb Voter Pro 1.0 ? :)
LMAO! :D |
| |
F7sus4
Registered: Apr 2013
Posts: 117 |
Quoting JammerYou make things bad with downright 1 but you need quite a lot of 1s so lonely vigilantes stand no bigger chance against cluster of other grades ;)
All calculation formulas are prone to generate some (positive or negative) bias in one way or another, but will be exploited only if there are people willing to do so.
On the other hand, there is no "perfect" system. Utilizing absolute (numeric) measures by people attributing arbitrary criteria generates bias, but so does the dispersion of the scale (10-point scales generate way more 8-9s for "very good" works, whereas in 5-point scales it is almost exclusively 5) because psychology, and so on, and so on. |
| |
Jammer
Registered: Nov 2002
Posts: 1335 |
Quoting F7sus4All calculation formulas are prone to generate some (positive or negative) bias in one way or another, but will be exploited only if there are people willing to do so.
On the other hand, there is no "perfect" system. Utilizing absolute (numeric) measures by people attributing arbitrary criteria generates bias, but so does the dispersion of the scale (10-point scales generate way more 8-9s for "very good" works, whereas in 5-point scales it is almost exclusively 5) because psychology, and so on, and so on.
Psychological bias can't really be avoided and it's IMHO not a system's job to compensate for it. It's easier and better to eradicate single malicious actions. If more people gave a prod weak grades, there certainly must be sth to it, obviously, and there's no need to fix it. This particular formula certainly doesn't aim for fixing biases, it just amplifies frequent votes and therefore assures stability of calculated result. |
| |
F7sus4
Registered: Apr 2013
Posts: 117 |
Quoting JammerPsychological bias can't really be avoided and it's IMHO not a system's job to compensate for it. It's easier and better to eradicate single malicious actions.
Note that this solution does nothing but implement a psychological factor into systemic compensation, as it is based on assumption that one or several votes were malicious/beneficial solely because they went against what most people said. That doesn't need to be true, and excluding them leads to self-confirmation bias, which is an equally rigged outcome. |
| |
Jammer
Registered: Nov 2002
Posts: 1335 |
Quoting F7sus4Note that this solution does nothing but implement a psychological factor into systemic compensation, as it is based on assumption that one or several votes were malicious/beneficial solely because they went against what most people said. That doesn't need to be true, and excluding them leads to self-confirmation bias, which is an equally rigged outcome.
That's right. But what interests me most is stable result, not quite people's thinking - whatever the bias comes out ;) Try it out:
Simple Average with ^2 Votecount |
| |
Frostbyte
Registered: Aug 2003
Posts: 183 |
Oops, sorry - that was me changing the values on spreadsheet. :)
The weighing seems to work reasonably well in lessening the effect of single or even a few downvotes. One change that would accompany this very well would be to show two decimals in the CSDB scores on the prod pages, not just in the top lists. |
| |
Jammer
Registered: Nov 2002
Posts: 1335 |
Quoting JojeliOne change that would accompany this very well would be to show two decimals in the CSDB scores on the prod pages, not just in the top lists.[/quote
All top prods would basically look like
All top prods would basically look like 9.99 :D |
| |
F7sus4
Registered: Apr 2013
Posts: 117 |
Quoting JammerBut what interests me most is stable result, not quite people's thinking - whatever the bias comes out ;)
Which might be a bad idea in the very beginning, as it is impossible to pull out reliable results from distorted input data. Therefore, the conclusion would not be to advocate for the improvement of the formula, but for the honesty of the votes. And to achieve that, transparency would be required.
I've tried your formula and while it provides more stable results with mixed input, it is definitely more prone to direct hate-vote (multiple 1s) when compared to the current CSDb system. But "it's only my opinion, I'm not an oracle" ;D |
| |
Jammer
Registered: Nov 2002
Posts: 1335 |
Quoting F7sus4it is definitely more prone to direct hate-vote (multiple 1s) when compared to the current CSDb system.
Of course every system is more or less vulnerable to mass attack of any kind. Review bombings on Metacritic can be quite a weapon at times :D
BTW, obviously I was wrong about result not being weighted. They are weighted on basis of vote count - what I meant is that results don't include fixed per-value weight. Just to be precise ;) |
| |
F7sus4
Registered: Apr 2013
Posts: 117 |
Quoting JammerReview bombings on Metacritic can be quite a weapon at times :D
Yes, this is completely true. Bombings are popular in big communities, but having enourmous vote-count also dispers the responsibility and the effect of a single participant. CSDb is not Metacritic, Rotten Tomatoes etc. In small communities being anonymous (or not) puts completely different weight on the decision-making process, and how each single vote/comment affects the outcome. |
| |
TheRyk
Registered: Mar 2009
Posts: 2241 |
You obviously suffer from Silly Season/Dog Days. Too hot to create music, but not too hot to vex your brain with Voting System :)
BTT/Krill's question: At least Jammer's formula could be easily hammered into few lines of BASIC Code and thus, perform way faster than Penisbruch stuff which seems to compute for ages. And the results do not seem to vary big deal.
PS: More often than not, even Jury votes do not vary so much from the infamous CSDb Down/Namevoting than people expect. Party voting is a totally different story, though. People who actually are there in persona, normally are voted more generously than remote entries, e.g. |
