data-science-workflow/Readme.md

# Data Science Workflow #

This repository explores through examples how to use the command line in an efficient and productive way for data science tasks. Learning to obtain, scrub, explore, and model your data.

# Introduction #
During this examples your will learn how to: (*i*) run docker containers, (*ii*) use the command line, (*iii*) run a basic application.

## Docker ##

Let us introduce docker, the first platform to make data science. Docker is a tool that allows developers, sys-admins or data-scientist to easily deploy their applications in a sandbox (**called containers**) to run on a host *operating system i.e. Linux, Windows, MacOS*. 

The key benefit of Docker is that it allows users to package an application with all of its dependencies into a standardized unit for software/data-science development. Unlike virtual machines, containers do not have high overhead and hence enable more efficient usage of the underlying system and resources.[^1]

## Installing and using a Docker image ##

In this case we are going to create a new Docker image to work with. The image is based on **Ubuntu Bionic:18.04** and it is created using `docker build`, by using the next `Dockerfile`:

``` shell
# Ubuntu
FROM ubuntu:bionic
ENV UNAME="data-science-workflow"
MAINTAINER gmarxcc
LABEL version="0.1"
ARG DEBIAN_FRONTEND=noninteractive
ENV LANG=C.UTF-8 LC_ALL=C.UTF-8

RUN apt-get update \
&& apt-get install -y python3 curl 

```
 and calling the `docker build` command:
 
 ``` shell
 docker build -t gmarxcc/workflow:0.1 .
 ```


### Docker Pull ###

``` shell
docker pull gmarxcc/workflow:0.1
```

### Docker Run ###

``` shell
docker run -it -v `pwd/data:/home `gmarxcc/workflow:0.1 
```

We recommend that you create a new directory, navigate to this new directory, and then run the following when you’re on macOS or Linux:

``` shell
$ docker run --rm -it -v`pwd`:/data datascienceworkshops/data-science-at-the-command-line
```


Or the following when you’re on Windows and using the command line:

``` shell
$ docker run --rm -it -v %cd%:/data datascienceworkshops/data-science-at-the-command-line
```

Or the following when you’re using Windows PowerShell:

``` shell
$ docker run --rm -it -v ${PWD}:/data datascienceworkshops/data-science-at-the-command-line
```

In the above commands, the option `-v` instructs docker to map the current directory to the `/data` directory inside the container, so this is the place to get data in and out of the Docker container.

### Testing the container ###

####  Command-line example ####

``` shell
curl -s http://gmarx.jumpingcrab.com/examples-data/76-0.txt | tr '[:upper:]'  '[:lower:]' | grep -oE '\w+' | sort | uniq -c | sort  -nr | head -n 10
```

#### Python example ####

**creating the factor operation on docker**

For testing the container a factorial function is created in Python 3: 

``` python
##!/usr/bin/env python3

def factorial(x):
    result = 1
    for i in range(2, x + 1):
        result *= i
    return result
```

Improving the factorial function with system: 
``` python
##!/usr/bin/env python3

def factorial(x):
    result = 1
    for i in range(2, x + 1):
        result *= i
    return result

if __name__ == "__main__":
    import sys
    x = int(sys.argv[1])
    print(factorial(x))

```

#### Nginx server example ####

``` shell
docker pull nginxdemos/hello
docker run -P -d nginxdemos/hello
docker ps 
```

# Command line basics #
This new way to work requires an overall understanding about we should call the command line, and according to [^2] it is mainly defined by:*(i)the command-line tools, (ii)the terminal, (iii)the shell, and (iv)the operating system*.

## The command-line tools ##
We use them by typing their corresponding commands on the *terminal*. There are different types of command-line tools, examples of this tools are: `ls`, `cat`, and `more` commands.

## Terminal ##
The terminal is the application where we type our commands in; see next figure:

![Terminal](https://i.blogs.es/56c9ee/cd/450_1000.jpg "terminal-ubuntu")

The dollar sign `$` shown in the figure is known as the *prompt*, and you are watching the typical *Ubuntu terminal* in version 18.04; other kind of prompts are `>`, `~`, `->`, among others.

The *terminal* is some kind of front-end to observe the input/output of a command process task.

## Shell ##

The third element is the shell. Once we have typed *a command-line tool* and pressed `<Enter>`, the terminal sends that command to the *shell*. The shell is a program that interprets the command. The image shows the Bash (Bourne Again Shell), but there are many others available like *Z shell*.


## Operating system ##

The last element is the operating system (OS), which is *GNU/Linux* in the Docker image. Linux is the name of the kernel, which is the heart of the operating system. The kernel has a direct contact with the CPU, disks, and other hardware. The kernel also executes the *command-line tools*. GNU, which stands for GNU’s not UNIX, refers to the set of basic tools. In this case the Docker image is based on Ubuntu Linux.

# Type of command-line tools #

The command-line tools are some kind of apps called by text and return text, strings or files. Each command-line tool can be one of the following five types according to [^2]:

  * A binary executable.
  * A shell builtin.
  * An interpreted script.
  * A shell function.
  * An alias.
  
The most common are the first two, while the others allow to build up a toolbox that will make us more efficient and productive.

## Binary executable  ##

Binary executables are programs in the classical sense. A binary executable is created by compiling source code to machine code. This means that when you open the file in a text editor you cannot read it[^2].
	
## Shell builtin ##

Shell builtins are command-line tools provided by the shell, which is Bash in our case. Examples include cd and help. These cannot be changed. Shell builtins may differ between shells. Like binary executables, they cannot be easily inspected or changed.

## Interpreted Script ##

An interpreted script is a text file that is executed by a binary executable. Examples include: *Python*, *R*, and *Bash scripts*. One great advantage of an interpreted script is that you can read and change it. E.g. a script `fac.py`. This script is interpreted by Python not because of the file extension .py, but because the first line of the script defines the binary that should execute it.

## Testing some tools ##

We employ the term command-line tool a lot, but so far, we have not yet explained what we actually mean by it. We use it as an umbrella term for anything that can be executed from the command line. Under the ho


# Notes #

- [ ] Make an container with Ubuntu 18.04 
- [ ] Packages to install: csvkit, 

[^1]: Docker for beginners, https://docker-curriculum.com/.

[^2]: Data science at the command line, 1st Ed,