6.6 KiB
Data Science Workflow
This repository explores through examples how to use the command line in an efficient and productive way for data science tasks. Learning to obtain, scrub, explore, and model your data.
Introduction
During this examples your will learn how to: (i) run docker containers, (ii) use the command line, (iii) run a basic application.
Docker
Let us introduce docker, the first platform to make data science. Docker is a tool that allows developers, sys-admins or data-scientist to easily deploy their applications in a sandbox (called containers) to run on a host operating system i.e. Linux, Windows, MacOS.
The key benefit of Docker is that it allows users to package an application with all of its dependencies into a standardized unit for software/data-science development. Unlike virtual machines, containers do not have high overhead and hence enable more efficient usage of the underlying system and resources.1
Installing and using a Docker image
In this case we are going to create a new Docker image to work with. The image is based on Ubuntu Bionic:18.04 and it is created using docker build, by using the next Dockerfile:
and calling the docker build command:
docker build -t gmarxcc/workflow:0.1 .
Docker Pull
docker pull gmarxcc/workflow:0.1
Docker Run
docker run -it -v `pwd/data:/home `gmarxcc/workflow:0.1
We recommend that you create a new directory, navigate to this new directory, and then run the following when you’re on macOS or Linux:
$ docker run --rm -it -v`pwd`:/data datascienceworkshops/data-science-at-the-command-line
Or the following when you’re on Windows and using the command line:
$ docker run --rm -it -v %cd%:/data datascienceworkshops/data-science-at-the-command-line
Or the following when you’re using Windows PowerShell:
$ docker run --rm -it -v ${PWD}:/data datascienceworkshops/data-science-at-the-command-line
In the above commands, the option -v instructs docker to map the current directory to the /data directory inside the container, so this is the place to get data in and out of the Docker container.
Testing the container
Command-line example
curl -s http://gmarx.jumpingcrab.com/examples-data/76-0.txt | tr '[:upper:]' '[:lower:]' | grep -oE '\w+' | sort | uniq -c | sort -nr | head -n 10
Python example
creating the factor operation on docker
For testing the container a factorial function is created in Python 3:
##!/usr/bin/env python3
def factorial(x):
result = 1
for i in range(2, x + 1):
result *= i
return result
Improving the factorial function with system:
##!/usr/bin/env python3
def factorial(x):
result = 1
for i in range(2, x + 1):
result *= i
return result
if __name__ == "__main__":
import sys
x = int(sys.argv[1])
print(factorial(x))
Nginx server example
docker pull nginxdemos/hello
docker run -P -d nginxdemos/hello
docker ps
Command line basics
This new way to work requires an overall understanding about we should call the command line, and according to 2 it is mainly defined by:(i)the command-line tools, (ii)the terminal, (iii)the shell, and (iv)the operating system.
The command-line tools
We use them by typing their corresponding commands on the terminal. There are different types of command-line tools, examples of this tools are: ls, cat, and more commands.
Terminal
The terminal is the application where we type our commands in; see next figure:
The dollar sign $ shown in the figure is known as the prompt, and you are watching the typical Ubuntu terminal in version 18.04; other kind of prompts are >, ~, ->, among others.
The terminal is some kind of front-end to observe the input/output of a command process task.
Shell
The third element is the shell. Once we have typed a command-line tool and pressed <Enter>, the terminal sends that command to the shell. The shell is a program that interprets the command. The image shows the Bash (Bourne Again Shell), but there are many others available like Z shell.
aqui voy
Operating system
The last element is the operating system (OS), which is GNU/Linux in the Docker image. Linux is the name of the kernel, which is the heart of the operating system. The kernel has a direct contact with the CPU, disks, and other hardware. The kernel also executes the command-line tools. GNU, which stands for GNU’s not UNIX, refers to the set of basic tools. In this case the Docker image is based on Ubuntu Linux.
Type of command-line tools
The command-line tools are some kind of apps called by text and return text, strings or files. Each command-line tool can be one of the following five types according to 2:
- A binary executable.
- A shell builtin.
- An interpreted script.
- A shell function.
- An alias.
The most common are the first two, while the others allow to build up a toolbox that will make us more efficient and productive.
Binary executable
Binary executables are programs in the classical sense. A binary executable is created by compiling source code to machine code. This means that when you open the file in a text editor you cannot read it[^2].
Shell builtin
Shell builtins are command-line tools provided by the shell, which is Bash in our case. Examples include cd and help. These cannot be changed. Shell builtins may differ between shells. Like binary executables, they cannot be easily inspected or changed.
Interpreted Script
An interpreted script is a text file that is executed by a binary executable. Examples include: Python, R, and Bash scripts. One great advantage of an interpreted script is that you can read and change it. E.g. a script fac.py. This script is interpreted by Python not because of the file extension .py, but because the first line of the script defines the binary that should execute it.
Testing some tools
We employ the term command-line tool a lot, but so far, we have not yet explained what we actually mean by it. We use it as an umbrella term for anything that can be executed from the command line. Under the ho
Notes
-
Docker for beginners, https://docker-curriculum.com/. ↩︎
-
Data science at the command line, 1st Ed, ↩︎