Data Science Workflow

This repository explores through examples how to use the command line in an efficient and productive way for data science tasks. Learning to obtain, scrub, explore, and model your data.

Introduction

During this examples your will learn how to: (i) run docker containers, (ii) use the command line, (iii) run a basic application.

Docker

Let us introduce docker, the first platform to make data science. Docker is a tool that allows developers, sys-admins or data-scientist to easily deploy their applications in a sandbox (called containers) to run on a host operating system i.e. Linux, Windows, MacOS.

The key benefit of Docker is that it allows users to package an application with all of its dependencies into a standardized unit for software/data-science development. Unlike virtual machines, containers do not have high overhead and hence enable more efficient usage of the underlying system and resources.¹

Installing and using a Docker image

In this case we are going to create a new Docker image to work with. The image is based on Ubuntu Bionic:18.04 and it is created using docker build, by using the next Dockerfile:

and calling the docker build command:

docker build -t gmarxcc/workflow:0.1 .

Docker Pull

docker pull gmarxcc/workflow:0.1

Docker Run

docker run -it -v `pwd/data:/home `gmarxcc/workflow:0.1 

We recommend that you create a new directory, navigate to this new directory, and then run the following when you’re on macOS or Linux:

$ docker run --rm -it -v`pwd`:/data datascienceworkshops/data-science-at-the-command-line

Or the following when you’re on Windows and using the command line:

$ docker run --rm -it -v %cd%:/data datascienceworkshops/data-science-at-the-command-line

Or the following when you’re using Windows PowerShell:

$ docker run --rm -it -v ${PWD}:/data datascienceworkshops/data-science-at-the-command-line

In the above commands, the option -v instructs docker to map the current directory to the /data directory inside the container, so this is the place to get data in and out of the Docker container.

Testing the container

Command-line example

curl -s http://gmarx.jumpingcrab.com/examples-data/76-0.txt | tr '[:upper:]'  '[:lower:]' | grep -oE '\w+' | sort | uniq -c | sort  -nr | head -n 10

Python example

creating the factor operation on docker

For testing the container a factorial function is created in Python 3:

##!/usr/bin/env python3

def factorial(x):
    result = 1
    for i in range(2, x + 1):
        result *= i
    return result

Improving the factorial function with system:

##!/usr/bin/env python3

def factorial(x):
    result = 1
    for i in range(2, x + 1):
        result *= i
    return result

if __name__ == "__main__":
    import sys
    x = int(sys.argv[1])
    print(factorial(x))

Nginx server example

docker pull nginxdemos/hello
docker run -P -d nginxdemos/hello
docker ps 

Command line basics

This new way to work requires an overall understanding about we should call the command line, and according to ² it is mainly defined by:(i)the command-line tools, (ii)the terminal, (iii)the shell, and (iv)the operating system.

The command-line tools

We use them by typing their corresponding commands on the terminal. There are different types of command-line tools, examples of this tools are: ls, cat, and more commands.

Terminal

The terminal is the application where we type our commands in; see next figure:

The dollar sign $ shown in the figure is known as the prompt, and you are watching the typical Ubuntu terminal in version 18.04; other kind of prompts are >, ~, ->, among others.

The terminal is some kind of front-end to observe the input/output of a command process task.

Shell

The third element is the shell. Once we have typed a command-line tool and pressed <Enter>, the terminal sends that command to the shell. The shell is a program that interprets the command. The image shows the Bash (Bourne Again Shell), but there are many others available like Z shell.

Operating system

The last element is the operating system (OS), which is GNU/Linux in the Docker image. Linux is the name of the kernel, which is the heart of the operating system. The kernel has a direct contact with the CPU, disks, and other hardware. The kernel also executes the command-line tools. GNU, which stands for GNU’s not UNIX, refers to the set of basic tools. In this case the Docker image is based on Ubuntu Linux.

Type of command-line tools

The command-line tools are some kind of apps called by text and return text, strings or files. Each command-line tool can be one of the following five types according to ²:

• A binary executable.
• A shell builtin.
• An interpreted script.
• A shell function.
• An alias.

The most common are the first two, while the others allow to build up a toolbox that will make us more efficient and productive.

Binary executable

Binary executables are programs in the classical sense. A binary executable is created by compiling source code to machine code. This means that when you open the file in a text editor you cannot read it[^2].

Shell builtin

Shell builtins are command-line tools provided by the shell, which is Bash in our case. Examples include cd and help. These cannot be changed. Shell builtins may differ between shells. Like binary executables, they cannot be easily inspected or changed.

Interpreted Script

An interpreted script is a text file that is executed by a binary executable. Examples include: Python, R, and Bash scripts. One great advantage of an interpreted script is that you can read and change it. E.g. a script fac.py. This script is interpreted by Python not because of the file extension .py, but because the first line of the script defines the binary that should execute it.

Testing some tools

We employ the term command-line tool a lot, but so far, we have not yet explained what we actually mean by it. We use it as an umbrella term for anything that can be executed from the command line. Under the ho

Notes

• Make an container with Ubuntu 18.04
• Packages to install: csvkit,

---

1. Docker for beginners, https://docker-curriculum.com/. ↩︎

2. Data science at the command line, 1st Ed, ↩︎