The Data Science Pipeline

The Data Science Pipeline


  • Planning
  • Data preparation
  • Modeling
  • Followup

Planning 

  1. Define goals
  2. Organize resources 
  3. Coordinate people
  4. Schedule project

Data preparation

  1. Get the data
  2. Clean the data
  3. Explore the data
  4. Refine the data

Modeling

  1. Crate model
  2. Validate model
  3. Evaluate the model
  4. Refine model

Followup

  1. Present model
  2. Deploy model
  3. Revisit model
  4. Archive assets


Comments

Post a Comment

Popular posts from this blog

Variables Types

Variables Types Quantitative variables Numerical, measurable quantities in which arithmetic operations often make sense. Continuous, could take on any value within an interval, many possible values. Discrete, countable value, finite number of values. Categorical or qualitative variables Classifies items into different groups Ordinal, groups have an order or ranking Nominal, groups are merely names, no ranking.  Examples Employees were asked to report their typical daily commute time, in minutes. What type of variable would their response be considered? Employees were asked to report their typical daily mode of transportation to and from work (i.e. Car, Bike, Bus, etc.). What type of variable would their response be considered? The company wanted to know how employees perceived the work of upper management. Employees were asked to report the satisfaction of upper management using a 1 to 5 scale (with the following representations: 1 - Extremely Unsat...
Read more

Scientific Method

Scientific Method The scientific method is based on systematic observation and   consistent logic.   Applying the scientific method increases our chances of coming up   with valid explanations.   It also provides a way to evaluate the plausibility of our scientific claims,   or hypotheses.   And the strength of the empirical evidence that we provide for   these hypotheses in our empirical study or research. Empirically testable Replicable Objective Transparent Falsifiable logically consistent Empirically Testable This means that it should be possible to collect empirical or physical evidence or observations that will either support or contradict the hypothesis. Replicable A study and its findings should be replicable. Meaning we should be able to consistently repeat the original study. If the expected result occurs only once, or in very few cases, then the result could just have been coincidental...
Read more

Confounder Variable

Confounder Variable  A confounder  is a variable that is related to both the independent and dependent variable, and partially,  or even entirely, accounts for the relationship between these two. An important thing to note about confounders, is that they  are not included in the hypothesis and they are generally not measured.  This makes it impossible to determine what the actual effect of a confounder was. The only thing to do is to repeat the study, and control the confounder  by making sure it takes on the same value for all participants. Example If we want to estimate house price and in our data set we have those variables : Size  Number of bedrooms  Address ZipCode Age of house  We might find that size is strongly correlated to the price of the house, but in our dataset, the size also correlated to the zip code. and if we loop deeper we might find that the size of the hou...
Read more