Description of a 'Review'

Post by **Tonie** » Mon Jun 19, 2006 4:37 pm

I think it would be a good idea if we devise a definition for the 'Reviews' for the extension site. This definition can act as a reference for developers, reviewers, and the review reviewers. The rules should IMHO consist of a number of things:

* What is the definition of a review?
* No support calls.
* Under what conditions are reviews moderated?
* Where can a person comment on a published review?

My question is, any ideas on this? Are there more topics that need to be covered?

PhilTaylor-Prazgod · Post by **PhilTaylor-Prazgod** » Mon Jun 19, 2006 4:45 pm

I wish I had kept my mouth closed

brian · Post by **brian** » Mon Jun 19, 2006 4:46 pm

#What is the definition of a review
- A report on a users experiences, including any difficulties they may have had and if they have found this extnsion to be better or
worse than a competing extension and stating the reason why

#no support calls
- Agreed a review is not the place to ask for support. However If

# Under what conditions are reviews moderated?
Apart from the obvious ones of bad language etc a review should be moderated if the review makes a statement that is known to be inaccurate. If at a later date the extension author contacts the moderator and explains satisfactorily that the review is wrong then the original review should then be moderated to avoi google etc storing the bad (and shown to be wrong) review.

A review should also be moderated if the reviewer is clearly biased against an extension because they have their own competing one or if the reviewer is felt to be connected in some way to the extension writer and therefore is making a review not based on experience etc.

#Where can a person comment on a published review?
I would not like to see comments on reviews but for there to be a method of extension authors to disucss with the editor a review as detailed above.

AmyStephen · Post by **AmyStephen** » Mon Jun 19, 2006 6:14 pm

I want to recommend one of the two approaches be utilized regarding comments:

1) No editing of any community submitted review < except for the obvious obscene, racist, criminal, personal attack remarks >;
-or-
2) Remove reviews completely.

These are community members who are reviewing. They are, for the most part, trying to help other people by providing a perspective. Adding a ton of rules to what they can or cannot say is not the solution, in fact, we are only prolonging this silly problem. The problem is that we will NEVER agree on a set of acceptable types of responses to what we think are "appropriate" opinions. And, I do not want this forum to try!

If I personally ask someone what they think of a component, they will likely tell me frankly what they thought. It may or may not be an accurate reflection. It could be dated -- an impression from awhile back. It could be they made a mistake that caused a component not to function. It could be that they absolutely detest the developer and are just trying to "get them." It could be that they do not like "commercial" open source -- or *worse yet* "closed source" (yikes!) and they are trying to influence my opinion in that direction.

The people who are trying to maliciously get someone will still try to maliciously get someone using any rule set you devise. Their attacks will only become more subtle and fit into your profile.

There are over 700 components out there. There is NO WAY this group of editors, no matter how many people you add to the staff, can install each component on every version of Joomla! it can run on, in every operating system possible, with every webserver, web browser and version, and configuration to test every single function available -- and tell people whether or not the comment someone made was appropriate.

THIS IS MADNESS to even try to edit these comments. The editing process will only diminish the community voice. It will not necessarily improve the overall quality of information. Those who are willing to work the editors will have more influence.

I believe that the third party provider can also add comments in the same manner as a community member. These can be used as "rebuttals" if so desired. I will read those as well!

I am an adult. I have my very own filtering mechanisms that work for me. I want the raw data or nothing so that I can reach my own conclusion. I do not want to wonder who talked to whom and how that might impact what I have to work with.

Third party providers - be forewarned - you are in a international space and you are providing product to a very large community of people, some of whom are stupid, some of whom are not nice, some of whom make innocent mistakes. It is HIGHLY LIKELY someone at sometime is going to say something - either publically through the Extensions site - or privately through email or a Skype call - that is not only damaging, but also inaccurate about your product.

Them's the facts.

Now, no one should be sorry they opened their mouth. This needed to come out and I, for one, am glad we are finally talking about it. Editing comments has bothered me for awhile.

Sorry, Tonie, I have so much respect for you, I feel bad that I did not answer the question as you have asked. I can only say - "unedited" or "do not collect" regarding community feedback on extensions. Thanks for collecting feedback; you always seem to know what to do. I am very impressed.

Extensions folks - I am not criticizing you, either. I think you are trying to be as fair and as generous and as helpful as you can be. Unfortunately, I really believe you are attempting to do a job that cannot be done and would prefer you not even try. But, I am hoping you do not find my comments discouraging and will continue to support your work regardless of what you choose to do. Me, I'd take my neck out of the noose, that's all.

Amy

Vimes · Post by **Vimes** » Mon Jun 19, 2006 6:52 pm

AmyStephen wrote:THIS IS MADNESS to even try to edit these comments. The editing process will only diminish the community voice. It will not necessarily improve the overall quality of information. Those who are willing to work the editors will have more influence.

Amy, you're missing the point.

A forum is a place where the public can discuss the pros/cons/failings of any given bit of software and the developer is quite within his/her rights to interceed at any point in the thread and agree/disagree with the poster. If I don't like what is said on the forums, tough, I won't bug anybody to take it off, I'll make sure that the events that lead to such an event didn't re-occur.

The Extensions server doesn't offer the same flow of conversation and as such is a special case.

Loathe that I am to admit it, I agree with PT. Developers should be allowed to ask editors to review reviews (sic). The extensions server has become my main source of business and if there ever were a time when I couldn't talk nicely to the editors and get a biased &/or malicious review "moderated" I too would take Jomres et al off it before you could blink. Better less business than no business at all. I gotta eat.

AmyStephen · Post by **AmyStephen** » Mon Jun 19, 2006 7:10 pm

Vimes wrote:The Extensions server doesn't offer the same flow of conversation and as such is a special case.

What about posting a rebuttal comment? I realize it's not a good environment for "conversation" but there is indeed at least a rudamentary mechanism to object to a comment. And, contacting the individual and working with them to have the comment removed is another solution.

And, neither one of you are giving us < the consumer > much credit! Do you really think, Vimes, that I cannot sort through these things? You have an excellent reputation, sir. I doubt I'd even read comments about your product -- you made it, it has to work. Period. And, if there are problems, I know because I have seen it - you work with people to resolve any issues.

Hello! Do you guys not believe in yourselves? Both of you -- what is wrong? We think you guys are great! Maybe we need a developer popularity and credibility measure so that people can use that -- and so that you guys get it. It is a bit shocking to me that you don't already know!

Vimes - there are developers who accuse the extension editors of being "buddies" with people and, therefore, are seriously editing comments in their favor. It is true! Why bother? Please trust your end user community. We are not complete morons. Do not use ME as a measure for everyone

Seriously - when I look at a comment someone makes, I look at who made the comment. If I have never heard of them, I put that comment in one pile - that pile has to SUM to something - I may or may not use that information in my decision. If it is someone known to me, it means more to me, but I might follow up with them for more information.

Also, I am probably going to download the damn thing and have a look of my own. Do you think we are so swayed by what one person says? Because, I am not!

Personally, I would rather know the process is free of undue influence. To me, that makes the data more valuable to use. Also, frankly, it takes good people out of the line of fire.

Vimes wrote: I gotta eat.

I certainly want you to eat. You do not look overweight - so please eat. Swing by, I will make you something to eat. You can join the family anytime for supper.

++++

One last suggestion that would also work for me -- each third party provider can decide whether or not to accept comments. And, if they accept them, they get what they get. If they do not want to take a chance with their end-user community, don't accept comments. That is a satisfactory compromise to me.

Thanks, Vimes, for your comments.
Amy

eyezberg · Post by **eyezberg** » Mon Jun 19, 2006 7:13 pm

brian wrote:#What is the definition of a review
- A report on a users experiences, including any difficulties they may have had and if they have found this extnsion to be better or
worse than a competing extension and stating the reason why

..

I would like to be able to say how much I hate visiting either the download link and or the website link for an extension only to find out info / demo / download is members only, when the extension is free. Why do I have to provide my email just to access information so I can actually at least imagine what this extension really does? Same for demos? Because it is the developers right, sure.. I would agree for downloads (bandwidth), but these could be hosted on the Forge.. but not for information about it.

Edit: Amy, just 2 things:
you'n'me, we hang around here too often, some others too, but not all people looking for a specific functionality and browsing extensions are as familiar with what exists and who's who!
you can't download and test commercial plugs (if there's no trial/demo)!

Vimes · Post by **Vimes** » Mon Jun 19, 2006 7:35 pm

To redress the sin of going OT again:

Tonie wrote:* What is the definition of a review?

A serious examination and judgment of something. (plagarised, I might add from Wordsearch.net).

Tonie wrote:* No support calls.

Agreed.

Tonie wrote:* Under what conditions are reviews moderated?

When the developer of the software being critiqued can provide proof that the accusations are unwarrented.

Tonie wrote:* Where can a person comment on a published review?

Here?

AmyStephen · Post by **AmyStephen** » Tue Jun 20, 2006 12:42 am

eyezberg wrote:Edit: Amy, just 2 things:
you'n'me, we hang around here too often, some others too, but not all people looking for a specific functionality and browsing extensions are as familiar with what exists and who's who!
you can't download and test commercial plugs (if there's no trial/demo)!

Huh? I have no idea who you are! Joe who? And, why are your hands blue? lol...ok. True - good points - but, there were more than TWO in there. I am certain of THAT.

Also, Joe, I don't know if you are saying that editing should continue or not. I am curious what you think.

++++

Vimes - the quote you gave that a review is "a serious examination and judgment of something" makes me think the real problem some of the third party folks might be having is that people who are offering reviews are not necessarily "qualified."

So, here's another idea -- this might sound ridiculous at first, but, I am totally serious.

Let the third party folks accept or reject their own comments. The software keeps tally - 100 comments submitted - 5 shown / 95 rejected or 99 shown - 1 rejected. The commenter should be allowed to modify their comment, too, so that rejections can be avoided. But, negotiations are made between the proper parties, no outsiders involved.

That a) gives me information and b) allows the third party to be 100% in charge and c) takes volunteers off the hot seat.

What about that?

< Tonie - I hope this is not considered OT, but rather a slight broadening of your design. If I am wrong, I apologize. Amy >

Vimes · Post by **Vimes** » Tue Jun 20, 2006 1:15 am

AmyStephen wrote:I am totally serious.

Let the third party folks accept or reject their own comments. The software keeps tally - 100 comments submitted - 5 shown / 95 rejected or 99 shown - 1 rejected. The commenter should be allowed to modify their comment, too, so that rejections can be avoided. But, negotiations are made between the proper parties, no outsiders involved.

Don't be daft, you're going down Moderated route that you wanted to avoid.

Look, let the server stay as it is, and when a developer thinks that a review is seriously out of order let the editing team look at the review twice. If the editors think that the developer has a point then it's pulled, else it stays.

Technology should never replace common sense, however much we want to abrogate responsibility.

AmyStephen · Post by **AmyStephen** » Tue Jun 20, 2006 1:51 am

Vimes wrote:Don't be daft, you're going down Moderated route that you wanted to avoid.

Daft
1. Mad; crazy.
2. Foolish; stupid.
3. Scots. Frolicsome.

Ok, dude, I choose "frolicsome", because those other choices - well, Vimes, those are "hurting words"

I believe the whole notion of editing community feedback is mad. It's downright ridiculous!

++++

The alure of this proposal, though, it that it is obviously and openly moderated in a known way. < Currently, I have no idea what kind of moderation took place before I started reading comments. That is what is not good IMO. >

This type of open moderation provides benefit to me as I look at the comments. The counts of rejected vs accepted reviews tell me something. Rejecting 95 out of 100 reviews tells me something is not working well. I do not need to know what. Rejecting 5 out of a 100, well, that is to be expected and does not spell trouble.

So, while it is indeed moderated, the moderation itself provides me information and more importantly keeps innocent people out of harm's way.

Vimes wrote:Technology should never replace common sense, however much we want to abrogate responsibility.

But, you are right and my idea would be difficult to implement.

++++

Vimes - let me ask you something, though. Why do the editors have to decide who is correct? The community member who posted a review or the disgruntled third party provider? The editors have to be judge and jury and choose? That is no position to be placed in.

And, unless these reviews are publically held or the results are posted, then the whole process is subject to legitimate critisim. What confidence do I have want as a consumer of this information? A secret judge, jury and evaluation process where community comments are changed? Like I said earlier, these complaints are already being leveled against this system! Fairly - of course not; but actually, yes.

Do you not at least see how this is not a good setup for anyone?

Vimes wrote:Look, let the server stay as it is, and when a developer thinks that a review is seriously out of order let the editing team look at the review twice. If the editors think that the developer has a point then it's pulled, else it stays.

Obviously, I do not get to make that decision and it is highly likely what you are proposing is exactly what will happen. Keep on - keepin' on is going to be the easiest thing to implement. But, understand, we are going to continue to have these situations we had today reoccur. I hate that, don't you? It just doesn't not work. And the sad thing is this -- it doesn't work for the volunteers. I hate that - it sucks.

OK. Thanks, Vimes. Amy

Post by **MMMedia** » Tue Jun 20, 2006 1:57 pm

I agree that factually incorrect statements should be corrected. But the slippery slope of the issue is where is it decided and who decides what is factually incorrect?

What I think everyone can agree on is that we all would like a fair and equitable system that allows both the users to give their experiences and the developers to receive meaningful feedback. Thus allowing the community to get the most of the excellent work that people are doing.

I think one way this could be done is allowing the developer the right of rebuttal. I also think that if a review is deemed such that it could be intentionally misleading or at its base untrue, that until a resolution is reached between the developer and the reviewer that the review be pulled pending resolution.

I think the developer would have to show a fairly high standard of proof to cause a review to be pulled, and it could only be used where rebuttal and resolution attempts have not been able to resolve the issue.

This would mean that a user could only review once. It would also mean that the developer could only rebut a review once. No going back and forth.

Reminicent of the feedback solutions used at a number of auction/commerce sites.

eyezberg · Post by **eyezberg** » Tue Jun 20, 2006 6:33 pm

1. requiere users to be logged with verified email, which is available to extension developper
2. allow review reporting
3. "notify" dev' of new reviews (dunno if this exists?)
4. notify reviewer of submission, rejection, and reporting!

this would help.

fathertime · Post by **fathertime** » Wed Jun 21, 2006 8:35 am

As a reader of reviews and customer, I think it is imperative that all reviews should be allowed. I do, however, have the following suggestions to improve the review system.

1) Allow the developer to tag 2 or 3 reviews placed in a separate block at the top of the review list. In other words, allow the developer to highlight several reviews he/she thinks best represent his/her product.

2) Instead of time-sorted reviews, sort by rating. Allow customers to see just positive comments as well as just negative.

3) Allow the developer to add a "read me" tag to comments which would link back to one single post that the developer could post clarifications and solutions on.

4) In regards to suggestion 3 - Ask that a potential reviewer please read the "read me" post before posting their comment.

5) Allow for anonymity of the reviewer, with the caveat that if the developer has chosen to add the "read me" tag to their review, they will be informed of this and allowed to adjust/remove their review.

Thank you for having this discussion. It is worth while.

Tim

Post by **Tonie** » Sun Jul 02, 2006 7:32 pm

I haven't forgotten this thread, have been busy watching football and helping out with the extra posts after the update to 1.0.10. The input so far is highly appreciated, hopefully I will be able to output a summary of guidelines later this week.

SKv · Post by **SKv** » Tue Jul 18, 2006 2:07 am

Take a look at this one: http://extensions.joomla.org/component/ ... /#rev-2757

This is not a 'Review'. It is support question. Please note that there is a link to bug tracker and a link to support forum.

I think that fathertime has given good ideas.

Vimes · Post by **Vimes** » Tue Jul 18, 2006 12:17 pm

It's swings and roundabouts, perhaps I shouldn't have approved that review, but it's difficult. The review contained all of the elements of a review and the output. As the wording isn't specifically a request for help then it could be considered a review with supporting evidence.

SKv · Post by **SKv** » Tue Jul 18, 2006 2:19 pm

I think that you did right: I see no reasons why not to approve this review. My argument is that developers should have chance to replay. I could help this guy.
Perhaps adding 'Support' button to top buttons section could help?

Post by **Tonie** » Wed Jul 19, 2006 12:37 pm

Thanks for all the input so far. We've discussed the topic as well in the extension team. At the moment, I have the following text. Some comments beforehand:

Why so many rules? I don't like too many rules, but the are a reference for both user and team member to know what is (not) allowed. Experience on the forum has taught us that issues get solved faster if people can relate to a list of rules/guidelines.

How was the list created? Current rules which are visible on the extensions site, this thread, issues team members had to deal with so far.

Static? These guidelines can and will change when the need arises.

Language? I'm proficient in English, but it's not my native language. If there are any mistakes in the text below, please point them out so I can change them

Extension site? There have been numerous changes to the extension site since it came online. Some of these were after requests from team members, some came from the community. Hopefully we can keep on improving current functionality.

===========

The Joomla! Extensions Directory is a free service provided for the Joomla! community. Any person or company is welcome to submit their extension for inclusion in the Joomla! Extensions Directory. There are some basic rules that all extensions must comply with to qualify for inclusion in the directory. Failure to comply may result in removal from the directory.

* All extensions must be installable on the Joomla! platform. The only exception to this rule are the applications in the 'Tools' category. This contains tools for creating and maintaining websites. They are not meant to be installed or change core files in Joomla!.
* Hacks which alter Joomla! source code must do so through the standard Joomla! install and uninstall process. Uninstalling should revert any changes made to Joomla! source code.
* Extension home and demo sites must not contain content of an adult or sexually explicit nature.
* Extension home and demo sites must not be promoting or engaging in illegal activities.
* Extension home and demo sites links must be to the site that is most relevant to the extension, links must not result in redirection to an alternative URL.
* Sites may not imply in any way that inclusion in the Joomla! Extensions Directory constitutes endorsement by Joomla! or Open Source Matters.
* Complete distributions of Joomla! will not be be listed.
* All content including reviews and extension descriptions must be in English.

Extension listings that infringe upon one or more of the criteria listed above will not be considered for inclusion in the Joomla! Extension Directory. In such cases the extension developer will be informed of the reason for the extensions rejection.
Any listings that is found to be in conflict with the rules will:

1. Be immediately deactivated (unpublished).
2. Developer will be informed of the reason and allowed 14 days to either rectify the problem or request removal from the directory.
3. If no reply is received in 14 days, the extension will be permanently removed from the Joomla! Extensions Directory.

Reviews

A review is a serious examination of a person's experience with an extension which is presented on the Joomla! Extensions Directory. Praise, difficulties, or comparisons with other extensions are all valid parts of a review, including the reasons why.

A review is not a location for requests for support. Support question should be taken to the Joomla! Forum or the developer of an extension. Please stay courteous and don’t use any offensive language. After finishing, a review will be accepted by a member of the Joomla! Extensions Directory team if it abides by the guidelines.

There is a possibility that a review contains a statement which is inaccurate. For example, this can happen if a new version of an extension fixes an outstanding issue, or if the statement isn't valid for another reason. We hold the right to withdraw a written review after it has been put online.

If you feel that a review contains incorrect information, please contact one of the moderators of the Joomla! Extensions forum. The Joomla! Extension Directory members will not moderate reviews, unless this is done in agreement with the reviewer. If a review poses a problem in some way, the review will be pulled offline. A short investigation will follow, both parties will be contacted by a member of the team. After investigating, it is both possible that the review is being kept offline, or that the review is put online again.

In no circumstances will personal information from Joomla! Extensions Directory members be shared with 3rd parties, unless this is the direct wish of said member.

Voting

* A user account can only vote once per extension. Please do not vote until you've used an extension thoroughly, making sure you have configured it correctly.
* It is not allowed for a any person to solely rate an extension to artificially inflate the rating or vote count.
* It is not allowed for the same person to have more than one account on the Joomla! Extensions Directory.
* It is not allowed for an extension developer to give negative votes to other extensions who are in competion for rating ranks, or to submit votes to extensions which offer similar functionality.

Any queries about these rules should be made in the directory forum.

brian · Post by **brian** » Wed Jul 19, 2006 12:53 pm

That all sounds very fair to me.

Perhaps one more suggestion (sorry only just thought of it)

If an extension is upgraded eg joomlaradio 1.0 to joomlaradio 2.0 Should this mean
a) a new extension entry is created and both co-exist
b) a new extension entry is created and the old one is unpublished
c) the exisiting entry is updated to reflect the change

Option a) would be my preferred solution as reviews etc are then current with the version being offered
Option b) means that all previous reviews,votes and ratings are lost - dont suppose that would be popular
Option c) existing reviews, votes and ratings may no longer be appropriate

Vimes · Post by **Vimes** » Wed Jul 19, 2006 3:17 pm

Good point. As we're currently using c) old reviews and votes can be mis-leading, but somebody who has had consistently good votes & reviews wouldn't want to lose them just because of an upgrade. It would discourage them from updating the server as and when.

It might be possible to associate votes & reviews with a version, but then you have the old problem "this extension already has lots of good reviews, why should I bother to add another one?" mentality. Older versions would have lots of good reviews, newer ones less, leading people to believe that newer versions weren't so good, and also anyway less updated components would look better than they should.

SKv · Post by **SKv** » Wed Jul 19, 2006 3:51 pm

I still think that adding support button along with download, homepage, demo, etc will help to avoid a lot of problems.

AmyStephen · Post by **AmyStephen** » Wed Jul 19, 2006 9:37 pm

Thanks to all of you for your work on this. Obviously, this is not an easy task and there are lots of different opinions. But, it is much appreciated that you called for comments, considered everyone's suggestions and I believe came up with a good process. Appreciate your volunteerism and dedication on behalf of the rest of us!

Amy

nant · Post by **nant** » Thu Aug 17, 2006 7:17 pm

Have been following this thread for a while ...

Reviews should be based on substantial user experience regarding the subject at matter. Doesnt have to be significant, but should be substantial. A reviewer that spends 10 mins on a 10,000 line component is unfair and does not help the user community form an opinion. Perhaps an additional field should be added to indicate the experience level the reviewer has regarding the item being reviewed.

Being on the CB team (that currently has 30+ reviews - http://extensions.joomla.org/component/ ... Itemid,35/) and browsing over the reviews it is obvious to me that users who invest some time are more likely to express better critisism and provide good feedback.

The Joomla! Forum™

Description of a 'Review'

Description of a 'Review'

Re: Description of a 'Review'

Re: Description of a 'Review'

Re: Description of a 'Review'

Re: Description of a 'Review'

Re: Description of a 'Review'

Re: Description of a 'Review'

Re: Description of a 'Review'

Re: Description of a 'Review'

Re: Description of a 'Review'

Re: Description of a 'Review'

Re: Description of a 'Review'

Re: Description of a 'Review'

Re: Description of a 'Review'

Re: Description of a 'Review'

Re: Description of a 'Review'

Re: Description of a 'Review'

Re: Description of a 'Review'

Re: Description of a 'Review'

Re: Description of a 'Review'

Re: Description of a 'Review'

Re: Description of a 'Review'

Re: Description of a 'Review'

Re: Description of a 'Review'