Chapter 4 Clarifying the Measurement Problem


Part II BEFORE YOU MEASURE

Meaning this is the Part which all the items you should consider before you actually go in to measure things. 



Chapter 4  Clarifying the Measurement Problem


Before you make a measurement, the first thing you ought to know is - WHATS THE DECISION

    What is the decision this measurement is supposed to support?

    If a measurement is not leading to some decision, why are you even measuring then?


Define the thing being measured in terms of observable consequences? - WHAT EFFECT IS BEING MEASURED

    Is the thing increasing or decreaseing ?

    Is more good or more bad?


How much do you know right now? - CURRENT STATUS OF KNOWLEDGE

    Empty your pockets


What will you do if this happens? or if that happens in measurement? - PROACTIVE DECISION MODEL

    Any threshold levels set on measurements, for respective activities


What is the value of additional information? HUNT ADD INFO OR NOT

    How important is it to get more info on this problem, before you proceed? Should you or should you not?


Measurement has no value if its not affecting a decision, or if its not being used to make a decision. 


A: Hey, I want to measure "Product Quality"

B: Why?

A1: Because I want to decide whether to change the ongoing manufacturing process or not..

B1: Okay, so at what level of product quality will you go for changing the manufacturing process and what do you mean by "Product Quality"?

A2: Because I want to commute management bonuses in quality program..

B2: Okay, so what is the formula for that calculation?


HTMA is my TENET,

This is a technology that no body else has right. its come from the future for me and me only.


Ofcourse everything needs to add up to an ultimate standardized solution, so we are doing the same here as well. **An universal approach** to Measurement -


1. Define a Decision problem and relevant uncertainities?

    1. What is your dilemma?

    2. Also define all the variables relevant to the dilemma

    3. Clear out the meaning of any ambigious terms like "Training Quality" or "Economic Opportunity"


2. Determining what you know now?

    1. First of all, you need to know how to express your uncertainities quantitatively, ie in terms of                 ranges and probabilities.

    2. This is a teachable skill, and you will go through calibration exercises in the coming chapters

    3. This "Checking your pocket" is important, because it tells us how much uncertain we are about             something, which then guides our next actions, and risks involved


3. Compute the value of additional information

    1. More information = less risk in decisions

    2. "Information Value" of a measurement is the value of that information you get from the                         measurement, that can help you to make good decision.

    3. Only engage in measurements that have high informational value


4. Apply the relevant measurement instruments/techniques to high value measurements

    1. Techniques like Random Sampling, Controlled Experiments etc,

    2. A lot on this in Chapter 9 to 13


5. Make a decision and act on it.

    1. In the end, when justifiable amount of uncertainity has been removed, it comes down to "Risk vs             Reward" situation and the decision maker has to make a decision.

    2. Risk Aversion, or the level of risk you are okay with, can also be calculated and filtered in, as we             will see in Chapters 11, 12 and 14.

    3. Making a decision in the end is part of this Cycle, and is not an element outside the realm of what         we are studying. 

    4. Tracking the results of your previous decisions, will also help a lot in making better decisions in             the future.


The Unexcepted Challenge of Defining a Decision

    1. Work out the decision Model first

    2. Don't measure just because something sound important

    3. Connect a decision to every measurement

    4. False Dichotomy decision - When one option of your decision is not really serious

    5. Impatience in clearing up the decision work


Dashboards and Decisions : Do they go together?

We have dashboards set up in every field and company, that give us min by min update on the measurements of certain important parameters we have predecided, but the author says the dashboards are really underutilized, because we dont attach a decision model to the dashboard.

For Eg, We don't know what to do if X falls below 10%, we will start scrambling when that actually happens and start brainstorming then, wasting valuable time. Instead we could have been a little proactive and worked out such contingencies before hand, so that our responses are immediate when it happens.


Decision Oriented Measurements : For Scientists too

The author worked with some scientists on a project, for which they build a decision model on an excel spreadsheet called INTERVENTION DECISION MODEL (IDM)


Requirements for a Decision

1. Every decision has 2 or more alternatives

    But it cant be a false dichotomy, alternatives must be serious

2. A decision has uncertainity. Best choice is not clear

3. There are negative consequences to making a wrong choice

    Even if both outcomes are positive, one would be more positive than other, so it is infact an         opportunity loss

4. A decision has a decision maker.


Potential Forms of a Decision

    1. Can be one big decision or several small decisions

    2. Can be discrete (Yes or No) or Continous (Optimal Slider)

    3. Can be done with one stakeholder or many, collaboraters, competetitors


If you can Understand it, you can model it

A decision is a well defined decision, if it can be modelled quantitatively. 


A Ridiculously Simple (But completely legitimate) Decision Model

1. Estimate the Cost of Action X

2. Estimate the Benefit of Action X

3. If Benefit of Action X > Cost of Action X, excecute Action X

You can further decompose the costs and benefits into more details if needed. 

This is a simple Cost benefit Analysis, which is being called a Decision Model here.


What Happens when you decompose the decisions?

1. You find that there are several other important variables that pertain to the judgement.

2. It turns out that merely decomposing highly uncertain estimates provides a huge improvements to estimates. THIS IS A TRUE FACT. The reason this happens is because your over estimates and under estimates cancel each other out. 


Your internal intuition itself is a type of decision model actually, just not a good one. It has a lot of inconsistencies, unstated assumptions and logical inference errors which is not visible to inspect. But quantitative models are not perfect either, they just dont have as many errors..


Great Stats Guy George Box once said rightly, "Essentially, all models are wrong, but some are useful"

So anything that can be modeled intuitively, can be modeled quantitatively.


"We find no sense in talking about something unless we specify how we measure it, a definition by the method of measuring a quantity is the one sure way of avoiding talking non sense."

- Hermann Bondi


Quantitative clarity is a strong foundation for the advancement of any field.


Definitions for Uncertainity and Risks

Uncertainity

The lack of complete certainity, that is, the existence of more than one possibility. The "true" outcome/state/result/ value is not known.

The way to measure uncertainity is by a probability assigned to possibilities. For eg. There is 60% chance this market will more than double in 5 years.


Risk

A state of uncertainity where some of the possibilities involve a loss, catastrophe or other undesireable outcome. 

The way to measure Risk is also by assigning probability to quantified losses For eg. We believe there is 40% chance the proposed oil well will be dry with a loss of $12 million in exploratory drilling costs.


Its all about thinking - not how you understand it, but how you will express it to someone else. 


Again repeating the set of questions author ususally asks when doing such AIE projects

1. What decision is this measurement for?

2. What do you mean by "IT security"?

3. What does improved IT security look like?

4. What would we see or detect that would be different if security were better or worse?

5. Furthermore, what do we mean by the value of security?


See how the annual cost has been Fermi decomposed for estimation


Answering in range is much easier.

Comments

Post a Comment

Popular posts from this blog

Chapter 6 Quantifying Risk through Modeling

Image
  Quantifying Risk through Modeling It is better to be approximately right than to be precisely wrong. HOW NOT TO QUANTIFY RISK Scoring method with all that colourful heatmap is utter bullshit. There is no evidence that it works and in fact there is evidence that it works in a bad way because it increases people's confidence in their risk management without actually managing risk. More on this perhaps in Hubbard's other book called "The Failure of Risk Management", which I will have to read someday. Insurance Premiums are good example of how rightly they quantify risk to a certain dollar value. Based on the level of risk you want to mitigate, your insurance premiums will be charged. They dont score risk in these instances as "Low" "Medium" or "High" LETS TALK ABOUT THE REAL SHIT - MONTE CARLO When you have a precise value of something it is easy to add, substract, multiply and divide them, but what do you do when you dont have a ...
Read more

Chapter 1 The Challenge of Intangibles

Image
  PEOPLE WHO GAVE US UNITS OF MEASUREMENT The book starts with a quote from Lord Kelvin, this is the guy who gave us the unit of measurement for Heat. I remember his weird experiment to measure heat from my school days. Its interesting that this guy is mentioned at the start of this book. In a way this quote is not just point to him, but everyone in the past who introduced any units of measurement. The whole concept of coming up with a unit of measurement requires a sophisticated understanding, so all these guys may be a potential mentor. Anyway the quote goes like... "When you can measure what you are speaking about, an express it in numbers, you know something about it; but when you cannot express it in numbers, your knowledge is of a meager and unsatisfactory kind; it may be the beginning of knowledge, but you have scarcely in your thoughts advanced to the state of science." Basically Einstein said, unless you can teach it, you have not really learnt it.. and Lord Kelv...
Read more

Chapter 3 THE ILLUSION OF INTANGIBLES: WHY IMMEASURABLES AREN'T

Image
  3 reasons why people think something cant be measured. They are rather misunderstandings. Concept - People misunderstand the concept of measurement itself. Once, you get it, you will say a lot more things are measurable than you thought before. Object - The things being measured is not very well defined. Sloppy and Ambiguous language is used. Method - Many people are simply not familiar with different methods to measure things. 3 reasons why people think something shouldnt be measured. Economic Objection - Too expensive to measure General Objection to meaningless of Statistics - You can prove anything with Stats. Circle Jerking BS. Ethical Objection - Dont wanna ask the hard questions needed to measure it. The Concept of Measurement Only an inexact man would call things exact. Measurement is not perfect nor certain. Defn of Measurement - Quantitaively expressed, reduction of uncertainity, based on one or more observations. Measurement is mere reduction of uncerta...
Read more