In this tutorial, we will provide basic examples of the conditional statements in bash scripting.

Passing Variables to Scripts via Command Line

In the script, we can define the variables with the dollar sign $. By adding a number, we are referring to the variable that we want to pass. So, the $1 is the first variable, the $2 is the second variable and so on. Let's see an example. Let's consider the following myexample.sh script.

echo "Hello, I pass the first variable $1 and the second one $2"

And let’s run it by typing:

bash myexample.sh var1 var2

Here, the var1 is assigned to $1 and…

Anna Karenina novel begins with the quote:

All happy families are alike; each unhappy family is unhappy in its own way. Leo Tolstoy

This is the Anna Karenina principle and personally, I have found it very suitable to describe the world of Data Science. Think about a Machine Learning pipeline:

1. Data Ingestion and Preparation
2. Model Training and Retraining
3. Model Evaluation
4. Deployment

We have started a series of articles on tips and tricks for data scientists (mainly in Python and R). In case you missed the previous installments:

Vol. 1:

Vol. 2:

Vol. 3:

Vol. 4:

Python

1. How to coalesce in pandas

This function returns the first non-null value between two columns:

import pandas as pd
import numpy as np
 
 
df=pd.DataFrame({"A":[1,2,np.nan,4,np.nan],"B":['A',"B","C","D","E"]})
 
df

In the first article, we learned how to start running Airflow in Docker, and in this post, we will provide an example of how you can run a DAG in Docker. We assume that you have already followed the steps of running Airflow in Docker and you are ready to run the compose.

Run the docker-compose

The first thing that you need to do is to set your working directory to your airflow directory which most probably consists of the following folders.

The simplest and fastest way to start Airflow is to run it with CeleryExecutor in Docker. We assume you have a basic understanding of Docker and you’ve already installed the Docker Community Edition (CE) on your computer.

Create an Airflow Folder

It’s convenient to create an Airflow directory where you’ll have your folders (like dags, etc.). So open your terminal, and run:

mkdir airflow-docker cd airflow-docker

I created a folder called airflow-docker.

Download the docker-compose.yaml File

To deploy Airflow on Docker Compose, you should fetch the docker-compose.yaml file. So let’s download it with the curl command.

curl -LfO 'https://airflow.apache.org/docs/apache-airflow/2.0.1/docker-compose.yaml'

Some directories in the container are mounted, which means…

We have started a series of articles on tips and tricks for data scientists, mainly in Python and R. (In case you missed them, here are Vol. 1, Vol. 2, and Vol 3.)

Python

1. How to get data from Google Drive into Colab

Google Colab is becoming more and more popular in the data science community. Working with Colab Jupyter Notebooks, you are able to mount your Google Drive so that you can get data from it directly. Let’s see how you can do that. You should open your notebook and type the following commands:

How to search Docker images from Docker Hub using the terminal.

Let’s say that we want to search for Python images. We can run the command:

docker search python

As we can see we get the NAME, the DESCRIPTION, the STARTS and if it is OFFICIAL and/or AUTOMATED. Always try to get images with many starts and prefer to be official.

How to get images from Docker Hub

You can get an image from a public repository by running the command docker pull and the name of the image. Let’s say that we want to get the “hello-world” image.

docker pull hello-world:latest