Levels of confidence

By cryptoref on Thu, 06/11/2020 - 17:25

Forums

So I'm trying to be much more careful about the words I'm using to express confidence in my reports. While I fully agree with the listing from 1.6, how do I inform the readers that I'm using that hierarchy? Appendix, footnote, ... I don't see other reports listing that, but not many readers will be able to tease out the hierarchy without a cheat sheet. I'm leaning towards an appendix.

Second, while I'm not likely not going to include the full text, just the hierarchy, with shorter descriptions to save space, I still need to cite EE. Don't think this is over three paragraphs, but I am requesting permission to include the hierarchy.

Submitted bycryptorefon Thu, 06/11/2020 - 19:39

Just reviewing the definitions in the hierarchy again. There is a circular reference. Probably is ...more likely than not. But that uses likely in the definition and it needs to be something else.

So for Probably how is this:

the author has some doubt regarding the assertion, but reliance on the assertion is [far] greater than the doubt

The far addition makes it stronger but it's still less that Certainly

David

Submitted bycryptorefon Thu, 06/11/2020 - 19:40

Sorry for being pedantic, but years of writing ISO standards has taught me that definitions are vitally important.

Submitted byEEon Fri, 06/12/2020 - 10:26

cryptoref, EE certainly agrees with you that we should choose our words carefully when we report the results of our research. As you say, it would probably be hard to find a research report that lays out the whole hierarchy of confidence levels used by the writer—except possibly, perhaps likely, as a footnote in those reports we submit to courts in legal cases. Apparently, you would feel more comfortable including a hierarchy in your reports to clients, in which case EE certainly would have no objection to your using the five definitions at 1.6, with attribution.  (Circularity? Definitely. <g>)

And yes, we’ll mull your suggestion.

Submitted bycryptorefon Fri, 06/12/2020 - 16:19

The current 1.6 hierarchy uses words that are just too close in meaning for me. It's hard to quickly see the difference between probably, likely, possibly, and apparently, all seem just nuances on the same meaning. So putting my standard writers hat on, here is a new hierarchy that i'd like to propose. Edits and comments are welcome.

  • Label                Confidence level           Description & Notes

  • Certain            Proved - no doubt        No reasonable doubt, based on sound research and good evidence, Research complete, GPS present
  • Confident       Proved - some doubt  Some doubt, based on sound research and good evidence, rationale overcomes doubt     Research complete, GPS present, rationale removes most doubt, but issues remain
  • Doubtful         Not proved - lots of doubt       Much doubt, based on sound research and good evidence, rationale unable to overcome doubt.  Research complete, GPS not present, rationale unable to resolve discrepancies
  • Insufficient    Not proved - insufficient evidence      Evidence, from sound research, unable to support assertion. Research complete or incomplete, unable to form GPS from known evidence
  • Believable     Untested - faith based    No evidence, but assertion follows from previously known proven assertions. Research nonexistent or incomplete, assertion is known to be true in other cases
  • Plausible      Untested - hypothesis     No evidence, no previous assertions, just experience applied to known facts. Research nonexistent, no previous experience with fact pattern, but research could provide evidence

The first three form the outcomes after completing research with the first two capable of creating a GPS. Insufficient can move up after additional research. The bottom two are when you are looking at the facts and making some sort of judgement. You have no facts, whatsoever. I wanted to call Plausible, WAG, but figured that wouldn't fly in a scholarly article :)

 

Submitted byEEon Sun, 06/14/2020 - 10:14

cryptoref, it is good to see a researcher thinking deeply about the words we use to report our findings and conclusions. That is the purpose of EE 1.6: not to prescribe words that must be parroted but to encourage thinking. To encourage researchers to differentiate between what is logically, reasonably proved and what needs considerably more support before it can be claimed. Your thoughts will help many others.

Submitted byscottwilds@aol.comon Sun, 06/14/2020 - 15:20

Hope you don't mind if I chime in here. We all try to assess where we are and whether we have met the GPS or not.

A question with cyptoref's suggested definitions is in the one called Confident. If "rationale removes most doubt, but issues remain" have we resolved conflicts? If not (some doubt remains), then the GPS is NOT met.

Submitted bycryptorefon Mon, 06/15/2020 - 15:57

Conflict and doubt are two different issues. Conflict arises when we have two pieces of information that give us different facts [for example, different birth years]. Doubt arises when our source(s) are not the strongest. Suppose I only have a census record from 1850 that says John Smith was in New York City. Any GPS i build using that census record is going to have doubts. No conflict, just insufficient direct evidence.

Further suppose that John's location in New York provides another thread that we are weaving to generate the GPS. It includes multiple facts, and here we are trying to indicate the strength, or from 1.6, levels of confidence. That single 1850 location, from a common name, is a very weak addition. It might be crucial, but trying to say, with a straight face, that we have certainty is not right. That's what 1.6 is trying to convey [Correct if I'm wrong EE], a citation is to allow the evaluation of the credibility of the evidence, our use of a qualifier indicates what level we, as the GPS creator, reached in our assessment.

As the GPS states:

Meeting the GPS neither requires nor ensures perfect certainty. Genealogical proofs - like accepted conclusions in any research field - never are final. Previously unknown evidence may arise, causing the genealogist to reassess and reassemble the evidence, which may change the outcome.

As I said up above, I come from the world of writing International standards [One of mine is ISO/IEC 27034 part 7]. It's important to those reading our reports to understand how confident we are in the evidence for each of the GPS we create. Dealing with events from a civil register, created at the time of the event, are very different from relying on birth information from a death certificate. While both enable the creation of a GPS, there are more doubts from the death certificate than there are in the birth register. We should include those doubts as we write our reports.

I'm actually thinking of adding one more level - Barely as in the GPS just does overcome the doubts.

Practical example.

GPS A - John is certainly the father. This implies no doubts.

GPS B - from the analysis, i am confident that John is the father. Check the analysis there is a lack of certainty.

GPS C - the analysis provides the bare minimum to assert that John is the Father. OK, the writer believes they've created a valid GPS but it's on the edge. I may come to a different conclusion after my review.

By providing these qualifiers, the reader instantly knows the confidence the writer had in the GPS. Conflicts are resolved, doubts may remain.