1.

Formulating a question that you can answer with data

The question-first approach tries to avoid the problems that can occur by finding spurious patterns in the data. Looking at the data first can be useful, but requires some rigour to avoid the dangers of data dredging where we misuse the data both to suggest hypotheses and then to confirm them.

This part considers; What is the question? Do we have (or can we get) the data answer it Could we use the answer once we have it?

There are three types of questions; What happened? This could be enough to inform future decisions given the assumption that the future will be like the past. What might happen? This allows us to make decisions given possible future circumstances. E.g., what will next year’s sales be if we add another store?

2.

Collecting and cleaning the data

This can include traditional ETL work into a data warehouse or data lake. It includes handling missing values through imputation, and normalization and scaling of data.

This can also include feature engineering. Feature engineering is the process of applying domain knowledge of the data to extract or synthesize more features. E.g., a timestamp field might be more useful if we extract the day of weekand the hour of day as separate fields if we are trying to model the expected ridership on public transit. E.g., the two fields lot width and lot depth might be more useful as a single derived field lot area if we are trying to predict real estate prices.

3.

Analyzing the data

We use unsupervised learning when labels (or example answers) aren’t provided. It is often used for discovering structure within data. Clustering finds groups of similar data. E.g., a clothing market segmentation problem might try to find three clothing sizes that are appropriate for the population’s combinations of height and weight. Clustering is also useful in document search applications finding a document “like” a given document. We use other unsupervised learning algorithms for visualizing high-dimensionality data in 2 or 3 dimensions.

Supervised learning assumes a labeled data set of examples. In the machine translation case, the Hansard provided the correct French/English answers for each example. Regression and classification differ in creating predictions that are real valued vs. categorical. E.g., Use regression to predict tomorrow’s temperature (real valued). Use classification to predict whether it will be sunny or not (categorical).

4.

Communicating the answer

This is the output of a Data Science project, sometimes called the Data Product. In some cases, the net result is a report or presentation. Some organizations emphasize repeatable analyses. In these cases, the report or presentation embeds the executable code used to create the analysis. These sorts of environments would likely use text-based documents in a version control system. This should be familiar to software developers. The analyses have to rely on repeatable data. So, these environments would also need the ability to recall exactly the same data set that was originally used. Another variation on reporting is an interactive application to allow the target audience to explore the data. This might be a custom application, or use of an off-the-shelf tool such as Tableau or Pentaho.

In other cases, a Data Science team might produce a running software module. This might be a prototype that a downstream software development team will productionize, or it might already be in a form useful for production.