trying to understand the star system

[brian]

Apologies if this has been asked/answered before but I'm confused about the star rating system

Today for example there is a brand new mambot with 1 vote that has 5 stars and other new extensions with no stars because they have no ratings yet.

Shouldnt there be a minimum number of ratings before something gets stars etc. Otherwise it is really has little value etc as the author of the extension can vote just once and get 5 stars 

[Tonie]

The number of stars is the average of all votes. Extensions with one vote can indeed get 5 stars.

An extension needs a set number of votes before it can be included in the top rated extensions. This means that extensions with one vote doesn't get into the top rated (if all is well ).

Is this what you wanted to know?

[brian]

Is this what you wanted to know?

Well yes it answers the question but it doesnt solve the problem.

From what you say I presume that their is a criteria for getting in the top rated extensions section. Cant the same criteria be applied to prevent a star rating on an individual item until it reaches those criteria.

Just seems kind of an anomaly to me that a brand new extension can be rated as 5 star without anyone other than the author haveing voted

[mpettitt]

The problem also goes the other way - people can put a low vote on a new extension without even bothering to try it, and hence predjudice people against using it.
Having a minimum requirement of, say, five votes before showing a rating at all would mean that new extensions get a chance to be tried before slipping into low star hell.

[brian]

hadnt thought of that but you are of course correct

[mpettitt]

I only noticed because I've published a new extension this morning, which hasn't been downloaded at all, yet someone has given it a one star rating. Had they tried it, fine, but rating something without bothering to download it is just stupid...

[gsbe]

I think that everywhere the star rating is displayed the number of ratings is also displayed. The user must weigh this data together just as they would when reading reviews.

The ratings and reviews are opinions and should be judged as such. We hope to provide enough data along with these opinions to help the user develop an informed opinion about these listings.

Brian- in the case you make you're discussing a newly listed extension. Since the ratings in the Extensions Directory are an average it can easily be said that the directory generally favors older listings. The directory overcomes its reliance on opinions over a period of time as more opinions are received. New listings do suffer a bit in this system but I believe this to be a good thing for the end user in the long-run. Taking the average of all opinions submitted over time should give us ratings that really mean something.

[brian]

I think that everywhere the star rating is displayed the number of ratings is also displayed. The user must weigh this data together just as they would when reading reviews.

Sorry but thats not right. the number of ratings are not displayed on this page http://extensions.joomla.org/component/ ... Itemid,35/

And on tis page it is the number of reviews not raitings
http://extensions.joomla.org/component/ ... Itemid,35/

Its only whne you get onto the details page like this one that you see the number of ratings
http://extensions.joomla.org/component/ ... Itemid,35/

[gsbe]

I see, I thought those were number of ratings but they are actually number of reviews. I also tried not to sound like the authority on this issue by saying that I thought that was what was happening because obviously I wasn't so sure.

Ok, do you think that we should change that? I'm hoping to hear opinions on the current system's validity as I explained it in my previous post.

[JoeJoomla]

I think there are three basic things users really want to know about an extension:

Will it predictably work with a particular version of Joomla as described without problems.
It is popular
What degree or level of difficulty is it to implement, manage, and maintain.

It's a lot to expect a star rating to give everything that's needed to know about an extension to every level of user. As long as it is made clear what the criteria really is for the star rating will the user then give it weight when making their decision. It's really easy to assume anything based on a glance of how many stars it has. I believe that when anyone, especially inexperienced users, see 5 stars that they believe that it is fully endorsed by the pros. If their experience is bad then they think the whole product is bad, not just the extension.

[gsbe]

Interesting thoughts, JoeJoomla. Are you essentially asking for it to be more obvious that these ratings and reviews are created by regular users and not officially endorsed by Joomla? Or would you like to change the current rating system to allow users to rate specificially only these three factors?

[JoeJoomla]

I think as long as it's easy to know what the criteria is, whatever it is, then people aren't making assumptions. I know that I'm very influenced by star ratings on websites that offer software.

Maybe someone else can think of other important factors that should be considered in a rating. If you could make it simple but highly relevant then you would have a great star system.

[brian]

How about just dropping the ratings.

The reviews are far more helpful and as long as anonymous reviews arent allowed are much less open to abuse.

No matter how big an excalaimer you have or how clear you make it that they are user reviews the potential for difficulties and confusion is still there.

Imagine the scenario where developer x starts to advertise his extensions as "rated 5star by joomla.org"

[JoeJoomla]

Imagine the scenario where developer x starts to advetise his extensions as "rated 5star by joomla.org"

"As seen in Joomla Magazine.com - 5 STARS!!"

Actually it might be a shame to remove the ability of someone to boast about their extension. It's certainly an attention getter.

[brian]

They still have the opportunity to be a "featured" extension

As you have said you are "very influenced by star ratings" and I am sure others are as well and it would be a real shame for joomla to be judged on the basis of skewed or inaccurate ratings.

[AmyStephen]

Everywhere the star rating is displayed the number of ratings is also displayed.

Yes. < I agree with Brian that those pages where it shows reviews instead of rating counts should be updated. Sounds like Graham that's what you thought, too. >

The ratings and reviews are opinions and should be judged as such.

Yes, we are adults - we are accustomed to listening to opinions and making up our own minds!

Since the ratings in the Extensions Directory are an average it can easily be said that the directory generally favors older listings. 

No, it does not "favor" them -- it increases the likelyhood of accuracy. Accuracy, in this case, being equal to the "average opinion" or "prevailing point of view."

The directory overcomes its reliance on opinions over a period of time as more opinions are received. 

Actually, over time, the directory is more like the opinion of the community. Which is *exactly* what we are attempting to measure. ?

New listings do suffer a bit in this system but I believe this to be a good thing for the end user in the long-run. Taking the average of all opinions submitted over time should give us ratings that really mean something. ;)

I just don't know how we would get around this -- anything new -- beit a new beverage or a new car or a new band -- whatever -- it takes awhile for a community to have experience with it and form an opinion.

Other than the one mistake pointed out -- where we are not listing the number of "votes" -- I cannot see anything wrong with this environment. To me, it's like the "open source", "closed source", "free" and "commercial debate." Please, don't take away my options just because you made up your own mind!

We are all adults here. I like to see the ratings. I like to read the comments. I like to know the # of downloads. But, in the end, these are all just some of many factors I am going to consider when I make my decision. If you don't like it -- ignore it!

@mPettit - I do not like to hear what happened to your extension. If you will PM me what it is, I will be glad to take an honest look at it and provide it a rating. I think that is really crummy. 
--> In fact - if I see a situation like this -- I am MUCH more likely to check out the tool to a) give them a chance if someone was unfair or b) out of curiousity. The WORSE thing that can happen to a new extension is to have NO ratings or reviews. Why bother?

just my 2 cents. Again, Graham - you guys are doing amazing things for us and it does not go unappreciated.

[Tonie]

Ok, Cye was really fast with this.

* An extension now needs five votes for the number of stars to be visible.
* The number of votes is also visible next to the extensions.

[mpettitt]

Now that's fast response!

5 stars for the Joomla Extensions team (with 1 rating) :-P

[brian]

Thanks Tonie

That makes much more sense

Brian