Chapter 5 Calibrated Estimates : How much Do you know Now?

 



  1. Define a decision
  2. FIND OUR YOUR CURRENT LEVEL OF UNCERTAINITY
  3. Figure the value of additional info

ABSURDITY

When estimating, even if you dont know the exact values, you still kind of know that some values on either end will be impossible or highly unlikely.

So the least you could do is, figure out the upper limit and the lower limit.

EMPTY YOUR POCKETS

Just like anything else in life, you need to know where you stand first to go where you want to. Quantifying your current level of uncertainty is a key part of the HTMA. It comes after defining the decision and before computing the value of additional information.

TENET

Start seeing the world differently, start seeing things as repetitive occurrences and probability distribution curves.

REFLECTION OF YOUR UNCERTAINITY

But every estimate should accompany with a CI. What CI does is it put out a reflection of your uncertainity in the equation and thats important to be put out there, if you want to use these measurements to describe reality.

FEEDBACK

To know how good we are at subjective probability assessment. We compare our expected outcomes to actual outcomes.

People get overexcited with the results and forget to feed it back to the model to improve the model. Very Pokerish Anne Duke.

WE OVERCONFIDENT

Vast majority of us are overconfident in our estimates, that our natural tendency (Answer is outside your predicted range). But Calling the odds is a learnable skill.

Talk about Professional Superpowers. Research has shown that oddsmakers and bookies are better at assessing odds of events than executives.

Calibration Exercise



Focus on what you do know.

As will all measurement problems, measuring your calibration can start with a question of the form "What would I expect to see if...?" In this case, we could ask "What would we expect to see if you were calibrated?"

Calibration Trick : Bet Money or Pretend to

One powerful tactic to become more calibrated is to pretend to bet money, but also introduce two different scenarios

Game A : You win $1000 if the true answer turns out to be between your upper and lower bounds. If not you win nothing.

Game B : You spin a 90/10 wheel with a dial. If the Dial lands on the 90 part, you win $1000, if it lands on 10% part, you win nothing.

QUESTION IS WHICH ONE WOULD YOU TAKE? A or B?

Its a trick question, because if you say your estimate has your 90% CI, then it should not matter to you which option you choose because both of them have a 90% chance of paying you.

This trick is called Equivalent Bet Test and it is done to make you realize that until you are indifferent to choosing any of these two options, you are not really 90% confident about your range.

HACKS FOR ESTIMATION

  1. Equivalent Bet Test

    1. Pretend you are winning money and put the question in the form of Game A and B.

  2. Repetition and Feedback

    1. Take Several tests in succession, assessing how well you did and try to improve

  3. Assume you are Wrong, Why do you thing you could be wrong?

Just assume your answer is wrong and then explain to yourself, if you were wrong, why would you be? This is to identify potential problems for your estimates. Inverse Thinking.

  1. Slip the range estimate to two binary estimates

Instead of estimating the range with upper and lower bound, you can ask yourself seperately,

Are you 95% sure it is over/under this amount? This cures a tendency for anchoring. Anchoring is when we pick a central value in our mind preemptively and +- some error to it to give an estimate.

  1. Absurdity Test

Ask yourself, what value seems to be ridiculously high or low and work your way in from there. This is the edge of our knowledge about that quantity.

Additional Tests in the appendix

REFLECTION OF YOUR UNCERTAINITY IN THE ANSWER

As experts in companies, we are taught to not give an quantitative value/ estimate because we may be held accountable to it.

But when you give a number range estimate, but also accompany that answer with CI, you are tying your uncertainities of the range to the range. And thats why CI is so important, to convey.

Also depending on your levels of CI, you might make a decision on whether to pursue additional information or not in the next step of HTMA.

The CI level you assign is for you only. Some other guy with more knowledge of the question, might assign higher CI probabllity. This is Bayesian.

When it comes to assessing your own uncertainty, you are the world's leading expert.

TIME TO GET CALIBRATED

How much time does it take to get calibrated?

4 rounds of 20 question, around 3.5 hours.


PHILOSOPHICAL INTERLUDE

We have not calculated this 90% CI. but its from the intution. Subjective CI is also CI in Bayesian System.,where the people decide how to live.

Frequentics might take an argument with calling Subjective CI as actual CI becaue it they are frequentist.

I am going to read this chapter for next 1 hour, whatever be the progress. So that I can enjoy better later.

Comments

Post a Comment

Popular posts from this blog

Chapter 6 Quantifying Risk through Modeling

Image
  Quantifying Risk through Modeling It is better to be approximately right than to be precisely wrong. HOW NOT TO QUANTIFY RISK Scoring method with all that colourful heatmap is utter bullshit. There is no evidence that it works and in fact there is evidence that it works in a bad way because it increases people's confidence in their risk management without actually managing risk. More on this perhaps in Hubbard's other book called "The Failure of Risk Management", which I will have to read someday. Insurance Premiums are good example of how rightly they quantify risk to a certain dollar value. Based on the level of risk you want to mitigate, your insurance premiums will be charged. They dont score risk in these instances as "Low" "Medium" or "High" LETS TALK ABOUT THE REAL SHIT - MONTE CARLO When you have a precise value of something it is easy to add, substract, multiply and divide them, but what do you do when you dont have a ...
Read more

Chapter 1 The Challenge of Intangibles

Image
  PEOPLE WHO GAVE US UNITS OF MEASUREMENT The book starts with a quote from Lord Kelvin, this is the guy who gave us the unit of measurement for Heat. I remember his weird experiment to measure heat from my school days. Its interesting that this guy is mentioned at the start of this book. In a way this quote is not just point to him, but everyone in the past who introduced any units of measurement. The whole concept of coming up with a unit of measurement requires a sophisticated understanding, so all these guys may be a potential mentor. Anyway the quote goes like... "When you can measure what you are speaking about, an express it in numbers, you know something about it; but when you cannot express it in numbers, your knowledge is of a meager and unsatisfactory kind; it may be the beginning of knowledge, but you have scarcely in your thoughts advanced to the state of science." Basically Einstein said, unless you can teach it, you have not really learnt it.. and Lord Kelv...
Read more

Chapter 3 THE ILLUSION OF INTANGIBLES: WHY IMMEASURABLES AREN'T

Image
  3 reasons why people think something cant be measured. They are rather misunderstandings. Concept - People misunderstand the concept of measurement itself. Once, you get it, you will say a lot more things are measurable than you thought before. Object - The things being measured is not very well defined. Sloppy and Ambiguous language is used. Method - Many people are simply not familiar with different methods to measure things. 3 reasons why people think something shouldnt be measured. Economic Objection - Too expensive to measure General Objection to meaningless of Statistics - You can prove anything with Stats. Circle Jerking BS. Ethical Objection - Dont wanna ask the hard questions needed to measure it. The Concept of Measurement Only an inexact man would call things exact. Measurement is not perfect nor certain. Defn of Measurement - Quantitaively expressed, reduction of uncertainity, based on one or more observations. Measurement is mere reduction of uncerta...
Read more