Natural Language Processing

A practical example of how to build several Machine Translation models in Python and Tensorflow

We will build a deep neural network that functions as part of an end-to-end machine translation pipeline. The completed pipeline will accept English text as input and return the French translation. For our model, we will use an English and French sample of sentences. We will load the following libraries:

import collections

Where helper.py is:

import os


A tutorial of how you can make interactive maps with folium and pandas

Folium provides a python interface for leaflet.js. Leaflet.js is a Javascript library for interactive maps and can be useful to know on its own. The benefit of using this library via Folium is that Folium makes it very easy to use from within a Jupyter Notebook and to access your python data structures (e.g. Pandas DataFrames). The documentation for Folium can be found on its official website. For this tutorial, we will work with the Baltimore Arrest Data.

Interactive Map with Baltimore Arrest Data

The map will include the following dimensions:

  • Age: In the form of isAdult where will be a pop-up in every observation
  • Race…


A walk-through example of Transfer Learning on Images for Classification problems

In this tutorial, we will provide you an example of how you can build a powerful neural network model to classify images of cats and dogs using transfer learning by considering as base model a pre-trained model trained on ImageNet, and then we will train additional new layers for our cats and dogs classification model.

The Data

We will work with a sample of 600 images from the Dogs vs Cats dataset, which was used for a 2013 Kaggle competition.

import tensorflow as tf
from tensorflow.keras.models import Sequential, Model, load_model
from tensorflow.keras.layers import Input, Conv2D, MaxPooling2D, Dense, Flatten, Dropout
import numpy as np
import os
import…


Did you know that Medium gives bonuses?

Image on Unsplash

Today, when I visited the “Medium Partner Program” to see my traffic and revenues, I saw this message:


How to apply an algorithmic trading strategy that takes an advantage of big whales movements

Image by Bitcoin News

In the cryptocurrency market, the big whales play a key role. These movements affect significantly the market. Take for example a well-known big whale, Elon Musk. Whenever he trades a big amount of Bitcoins, the market follows him. In this post, we will try to build an algorithmic trading strategy that takes an advantage of big whales' movements.

The Scenario

We can detect the whales from the per-minute volume. When there is a significant spike in the volumes, it means that there is an anomaly and most probably is due to the trades of a big investor, the so-called “whale”. Usually, the…


Example of Programming Quizzes in Python

Image in Unsplash

It is common to be asked to build some functions in Python (or any other programming language) during interviews in order to assess your coding style, your skills and your way of thinking. Let’s provide some examples:

Assume that we have a sequence of N rolling Die results, taking values from 1 to 6. Assuming that your input will be a sequence of results like “56611166626634416” build the following functions:

Question 1

How many times did it occur in the trial, that exactly two 6s were rolled after each other? …


Image by Predictive Hacks

We have started a series of articles on tips and tricks for data scientists (in Python and R). In case you have missed:

R

1.How To Remove The Correlated Variables From A Data Frame

When we build predictive models, we use to remove the high correlated variables (multi-collinearity). The point is to keep on of the two correlated variables. Let’s see how we can do it in R by taking as an example the independent variables of the iris dataset.

Get the correlation matrix of the IVs of iris dataset:

df<-iris[, c(1:4)] cor(df)

Output:

Sepal.Length Sepal.Width Petal.Length Petal.Width
Sepal.Length 1.0000000 -0.1175698…


A walk-through example of how you can get a “Sentiment Score” for each word by applying the Naive Bayes theory

We have provided an example of how to get a sentiment score for words in Python based on ratio frequency. For this example, we will work with the Naive Bayes approach taking into consideration a Twitter dataset that comes with NLTK which has been manually annotated. The sample dataset from NLTK is separated into positive and negative tweets. It contains 5000 positive tweets and 5000 negative tweets exactly.

Positive and Negative Probability of a Word

We have provided an example of Naive Bayes Classification where we explain the theory. …


A walk-through tutorial of how you can apply Naive Bayes Classification in NLP tasks from scratch.

Image on Unsplash

In this tutorial, we will get the Bayesian Score of each word as well as of the whole Subject Line. The score will indicate the chance of a Subject Line and/or token being “spam”. You can find the dataset here. We have used the same dataset, in the Email Spam Detector Tutorial, so feel free to compare the Bayesian approach with the Logistic Regression.

Load the libraries

import pandas as pd 
import numpy as np
import re from collections
import Counter import string

Theory and Formulas

So how do you train a Naive Bayes classifier?

  • The first part of training a naive Bayes…


How to get the Association Rules in Python

Image on Unsplash

We have provided a tutorial of Market Basket Analysis in Python working with the mlxtend library. Today, we will provide an example of how you can get the association rules from scratch. Let's recall the 3 most common association rules:

Association Rules

Given a set of transactions, find rules that will predict the occurrence of an item based on the occurrences of other items in the transaction. For example, we can extract information on purchasing behavior like “ If someone buys beer and sausage, then is likely to buy mustard with high probability

Let’s define the main Associaton Rules:

Support

It calculates…

George Pipis

Data Scientist @ Persado | Co-founder of the Data Science blog: https://predictivehacks.com/

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store