On Reputation Again

I did not have time yesterday, so I skipped straight to an English article, and ZIGOROu kindly made a Japanese translation. Many thanks (・人・).

I will make a Japanese version as well. It would be far too unhelpful to provide only the English version.

A Miscellaneous Note on Reputation

I had better write this down before I forget.

1. A reputation must include an identifier for the subject being rated.
2. The same applies to an identifier for the rater.
3. It must clearly state what was rated.
4. For multiple reputations to be aggregatable, the distribution of each reputation and the resulting distribution after they are combined (as with normal distributions) must be known.
5. Along with the rating, information about its distribution (the type of distribution, mean, standard deviation, and so on) must also be published.
6. The displayed score must have a meaning that ordinary people can understand intuitively.
7. Date rated
8. Rater’s signature

Accordingly, a file containing a score includes the following:

The table above proposes the cumulative distribution P(X<x) as the displayed score. Its meaning should be clear to anyone. If the displayed score is 95.5, the subject has achieved an excellent result placing them in the top 5% under that evaluation criterion.

As described in the OpenID TX proposal, it also includes public-key information.

By using this, someone attempting to communicate with the rated subject can confirm that the other party is the same entity that received the rating.

This data is serialized in formats such as XML, JSON, or even tag=value.

So I will use this as another informational input to the ORMS TC at OASIS Open.

The following are comments on ZIGOROu’s article.

So is this basically a bullet-point memo about how the ratings exchanged within the OpenID TX proposal (Trusted Data Exchange) concept should be created and what kinds of values they should have?

Yes. At the same time, Abbie Babir assigned me the task of writing a White Paper as the first step in discussions at the Open Reputation Management System Technical Committee being launched at OASIS Open, and this is part of it.

Put simply, I think it means this:
Each participant that appears in the process of authenticating an identity needs to be evaluated.
In OpenID terms, that means the OP, RP, and End User each need an evaluation.
The people who provide those evaluations also need to be evaluated.
Incidentally, OPs, RPs, and End Users can themselves be evaluators, and I do not think the evaluation system described below is limited to those participants. I seem to recall the Trusted Data Exchange saying something along those lines.

Yes, exactly. It is not stated in this memo itself, but anyone can evaluate anyone else, and evaluators are also evaluated. This is not a model in which there is one correct person at the center who issues absolute evaluations.

There should be multiple evaluators (a distributed evaluation system), and ultimately an evaluation should be reached that considers all the ratings, like an average score.
The evaluators should also evaluate one another and disclose their ratings to one another.
That is how I understand it.

The “distribution” mentioned in the original text means a probability distribution. I expect that one will combine evaluations created in many places to produce one’s own evaluation, but when aggregating them, unless both the distribution of each evaluation and the distribution after aggregation can be calculated, it becomes unclear what is actually being aggregated.

While I think this is a wonderful approach, what I still do not really understand is:
What do evaluators other than OPs, RPs, and End Users evaluate first, and according to what criteria?

OPs, RPs, and End Users can also evaluate one another, but do the values produced by feeding back those evaluations become the ratings provided by such evaluation services?

The evaluation criteria will probably vary. At least initially, I expect there to be audit ratings of the operational quality of OPs and RPs, and financial ratings of the operating companies.

For OPs, something like an audit seems necessary, including an audit of the system-related aspects.

I think the same applies not only to OPs but also to RPs.

As far as I can tell from the Trusted Data Exchange overview, both the RP and OP query a Reputation Service during the sequence of steps through which an End User’s Claim becomes a Verified Identifier.
Considering points like that, I wonder what indicators ultimately affect each party’s score.
I have high hopes for this kind of framework and am very interested to see exactly how it will be implemented.

With a community-based rating method (incidentally, rating methods and algorithms are outside the scope of the ORMS TC), matters such as who votes and how they do so are quite important.

I suppose I need to make a Japanese translation of the TX document too…

(The following was added in response to ZIGOROu’s comments.)
Incidentally, I deliberately left this out of the article because it is written in TX, but the point is simply that audit-based reputation services will probably emerge because they have business potential; there can also be various community-based services. The party making a decision can consider all of these together. That is why probability distributions and related matters are discussed… A reputation service could scan systems with Nessus every day and use the results as its score.

As ZIGOROu noted, assigning a score to the PrivacyMark is a good approach, and someone will probably do that as well. Obtaining a PrivacyMark is fairly expensive, though…

Incidentally, the RP rating is merely a reference for users when providing information. If you know the other party well, you can of course ignore the rating.