# Video 5 (3 A's)

## How do you know whether to make progress?

### Before lean (Vanity metrics)

 - Just do stuff
 - Vanity metrics
    - Look at total no question answered
    - Total number of chat messages
 - Arguing / opinions about technical things that go nowhere


### After lean (Split testing)

 - Look at real data like conversion
 - Engineer X can put feature Y in split testing and see real data. Real data can show negative / positive bonuses

## 3 A's

 - Actionable. Based on results from testing you can take the correct action. It should be obvious what action you should take as the result of a test
 - Accessibility. Reports from split testing should be easy to read.
 - Auditable. Every feature has a report card of what matters. "We shipped this feature but it didn't improve the business". "Metrics are people too". The report must represent real people

    if positive
        then do action
    if data is accessible
        then we see that it is positive
    if data is audible
        then we know there are real customers behind it
        then if data is negative we kill project

Audible test kill pet projects of no value.

## How to integrate / do framework for split testing

 - Easy reporting, should be obvious. Should be readable.
 - Hackathon day where everyone adds a split test
 - team meeting with report data

If you come up with a split test project. If it wins the data it stays, if it loses it leaves.

Less risk of trying a feature as we can tell whether it's good or bad

## Metrics

 - We have 6 to 10 criteria we run our business on
    - converting to paid
    - signing up
    - come back tomorrow
    - come back next week
    - learn something
 - We have key metrics
    - We want you to learn. Do they learn?
    - signing up
    - user generated distribution
    - converted to premium
    - engaged in parts of the system that leads to learning

Each metric is of the form "Of the people that are in the test, what fraction of people did this thing"

## Questions

 - Do we argue / throw opinions about things that we can solve by doing A / B testing based on data?
 - ---Do users get annoyed with our split tests---
 - When doing a split test that we expect to have negative result. THE VALUE OF LEARNING THE RESULT IS WORTH THE NEGATIVE EFFECT ON YOUR USERS

## Extreme programming

 - pair programming
 - agile development
 - test driven development
 - pivotal tracker. Getting things done for work
    - user experience story
    - developer does what is obvious

## Advantages

Don't talk / present / play politics. Build and bring data!
Reduces risk / fear of trying something

# BUILD AND BRING DATA. DONT PRESENT / BANTER / POLITICS

## Fear of customer seeing weird stuff

Only one time someone emailed us and "hey why does your price keep changing". It's not that big a deal. Nobody cares. Nobody knows

# DON'T ASK FOR PERMISSION ASK FOR FORGIVENESS

## What do you really need to build to test this thing?

## The bigger the test the more learning you get

Test big differences and optimize later. Do far left vs far right then binary search to optimum point

# IF IT DOESNT LOOK GOOD AND PEOPLE STILL USE IT THEN THERE IS REAL CORE VALUE THERE.

# MAKE FUNCTIONAL FIRST THEN BEAUTIFUL

# A GOOD DESIGN IS ONE THAT CHANGES CUSTOMER BEHAVIOUR IN A QUANTIABLE WAY