README.md

README.md

Tiew Kee Hui

Hi, I'm Kee Hui, a software engineer based in Singapore. I build full-stack applications, recommendation systems, and data & machine learning pipelines.

Current focus

I am now working at Open Government Products building for public good.

Full stack
Recommendation systems
Data engineering

Technical stack

Python
SQL
Java
Go
TypeScript
PyTorch
PySpark
pandas
scikit-learn
MLflow
FastAPI
BentoML
Flask
Spring Boot
Angular
Airflow
Docker
Kubernetes
Helm
Terraform
Kafka
GCP
AWS
BigQuery
Redshift
Postgres
Redis

Experience

Download PDF

Open Government Products

Senior Software Engineer · September 2025 to Present

Fullstack software engineer building for public good.

ShopBack

Senior Software Engineer (Machine Learning) · June 2022 to August 2025

Developing recommendation systems across 11 countries, maintaining FastAPI and Kafka services handling about 3 million requests per day, and improving P99 latency by 50% through service migration.

MoneyLion

AI Engineer · March 2021 to April 2022

Integrated Amazon Personalize recommendations, built Kafka Streams interactive query APIs, developed sampling pipelines, and optimized Aurora RDS to S3 exports from hours to minutes.

AirAsia

Data Scientist · September 2019 to February 2021

Built customer spending dashboards, deployed Airflow and JupyterHub, moved hundreds of gigabytes across systems, and contributed to experimentation infrastructure with Presto and Metabase.

iFAST Capital

R&D Software Engineer · September 2018 to August 2019

Developed a customer-service chatbot using AngularJS, Spring MVC, Python, and AWS; experimented with BERT, Rasa, GloVe, word2vec, and fastText; and conducted churn analysis with XGBoost, Random Forest, and SHAP.

Writing.app

Technical notes and engineering writeups

Code-heavy posts, data explorations, visualization experiments, and machine learning notes.

Shutting Down Python's ThreadPoolExecutor

To cancel pending futures during shutdown of ThreadPoolExecutor for Python 3.7, 3.8 and 3.9

Jul 10, 2021 10 min read

Populating MySQL Docker Image

A tutorial on starting up a MySQL Docker image on your local machine populated with some dummy data for local testing

Dec 11, 2020 1 min read

Tensorflow 2.0 / Keras - LSTM vs GRU Hidden States

Exploring cell state and hidden state for LSTM and GRU while doing Tensorflow's Neural Machine Translation with Attention tutorial.

Jun 25, 2019 4 min read

Editing Newest JSON File In A Directory

This post shows how to edit the newest (by create date) JSON file from a Windows directory

Jun 21, 2019 2 min read

Reinforcement Learning: Deep Q-Network (DQN) with Open AI Taxi

Python code in a Jupyter notebook to implement Deep Q-Network (DQN) using the Open AI Taxi environment and Keras-RL

Jan 22, 2019 1 min read

Reinforcement Learning: Q-Learning with Open AI Taxi

Python code in a Jupyter notebook to implement Q-learning using the Open AI Taxi environment

Jan 20, 2019 1 min read

Visualization of Total Accidents in Malaysia

This choropleth map about road accidents in Malaysia was coded using D3.js as part of fulfilling the Data Visualization course in Multimedia University.

Feb 19, 2018 1 min read

Visualization of Crimes in the USA

These visualizations for the FBI Crime dataset were coded using D3.js as part of fulfilling the Data Visualization course in Multimedia Unviersity. Contributors to this project are Tiew Kee Hui (me) and Lee Bing Yuan.

Feb 19, 2018 1 min read

Association Rule Mining on the Extended Bakery dataset

Association rule mining exercise conducted on the Extended Bakery dataset. This post includes the motivation of this exercise and the R code to determine which products to sell together based on association rule mining.

Jan 30, 2017 5 min read

Exploratory Data Analysis on the Student Alcohol Consumption dataset (Code)

Continuation of the exploratory data analysis exercise on the Student Alcohol Consumption dataset. This post includes the R code to answer the questions and relationships posed previously in the Introduction post.

Dec 31, 2016 1 min read

Exploratory Data Analysis on the Student Alcohol Consumption dataset (Introduction)

Basic exploratory data analysis on the Student Alcohol Consumption dataset obtained from UCI Machine Learning. This post contains the justification on exploring certain variables and interesting relationships in the dataset.

Dec 30, 2016 13 min read

Article.md

Shutting Down Python's ThreadPoolExecutor

To cancel pending futures during shutdown of ThreadPoolExecutor for Python 3.7, 3.8 and 3.9

July 10, 2021 10 min read

ThreadPoolExecutor is the recommended way to perform multithreading in Python. Multithreading is a way to speed up your application and is suitable for I/O bound tasks such as HTTP requests, database operations, etc.

Introduction

The simplest way to show how to use ThreadPoolExecutor is by example:

from concurrent.futures import as_completed, ThreadPoolExecutor

def task(num):
    return f"I'm running in a thread: {num}"

futures = []
with ThreadPoolExecutor(max_workers=2) as executor:
    for i in range(1, 11):
        futures.append(executor.submit(task, i))

    for future in as_completed(futures):
        print(future.result())

The application above will print the statement “I’m running in a thread: ” with the number 1 to 10, which will most probably be out of order.

Code Explanation

First, we created a ThreadPoolExecutor with 2 threads. Then, we submit 10 tasks to the ThreadPoolExecutor with a for-in loop, and in turn receive Future objects which we store in a list. We then loop through the list of futures using the as_completed() function, which allows us to receive futures that complete first instead of going sequentially through the futures list. Finally, we obtain the result of the task() function by calling future.result() and print the outcome.

Handling Exceptions

The example above was super simple, but what if we need to do something more complicated like firing concurrent HTTP requests. There is a possibility that the code may hit Exceptions during runtime due to uncontrollable reasons i.e. network issues, configuration issues. We can put a try-except block to handle Exceptions, but what if we want to cancel all submitted tasks when a future fails?

Shutdown

There is a built in function for ThreadPoolExecutor called shutdown().

In Python 3.7 and 3.8, shutdown() only stops the ThreadPoolExecutor from accepting new tasks. This means that if we submit all our tasks in one go at the beginning, and mid-way through a task fails which causes the future to return an Exception, other pending tasks that have not started running and are still in the executor’s job queue will NOT be cancelled.

Fortunately, it was revamped in Python 3.9 to allow users to cancel pending tasks in the executor’s job queue.

Python 3.7 and 3.8

At the time of writing this blog post I was using Python 3.7.10 and Python 3.8.5.

from concurrent.futures import ThreadPoolExecutor

with ThreadPoolExecutor(max_workers=2) as executor:
    executor.shutdown(wait=True)

In Python 3.7 and 3.8, shutdown() only accepts one boolean parameter, wait. When wait = True, Python will wait until all submitted tasks have finished running before shutting down the ThreadPoolExecutor.

When wait = False, it will still behave in the same way. The only difference is that the call to executor.shutdown() will not block. The executor will continue running all pending tasks, but will not accept new tasks and will completely shut down after all tasks are ran. Meanwhile, the main thread continues with our script’s main execution flow.

from concurrent.futures import ThreadPoolExecutor, as_completed
from datetime import datetime
import time

def task(num):
    if num == 5:
        raise ValueError("Num is 5")
    else:
        # To ensure this task doesn't complete too fast
        time.sleep(1)
    return f"I'm running in a thread: {num}"

futures = []
with ThreadPoolExecutor(3) as pool:
    futures = [
        pool.submit(task, i) for i in range(1,10)
    ]

    for future in as_completed(futures):
        try:
            n = future.result()
            print(f"{datetime.now()} - {n}")
        except ValueError as e:
            print(f"{datetime.now()} - EXCEPTION! {e}")
            pool.shutdown(wait=True)

print(f"{datetime.now()} - Run complete")

In the code example above, a ValueError exception will be raised when num = 5. After hitting the ValueError exception, the script will call shutdown().

When wait = True, Python will wait for all futures after num = 5 to complete and print it out (almost) all at once.

2021-07-10 11:42:15.382637 - I'm running in a thread: 1
2021-07-10 11:42:15.383645 - I'm running in a thread: 2
2021-07-10 11:42:16.389649 - I'm running in a thread: 4
2021-07-10 11:42:16.389885 - EXCEPTION! Num is 5
2021-07-10 11:42:18.397034 - I'm running in a thread: 3
2021-07-10 11:42:18.397034 - I'm running in a thread: 7
2021-07-10 11:42:18.398046 - I'm running in a thread: 6
2021-07-10 11:42:18.398046 - I'm running in a thread: 9
2021-07-10 11:42:18.399038 - I'm running in a thread: 8
2021-07-10 11:42:18.400030 - Run complete

Since wait = True, after handling the ValueError exception, the main thread will wait for all tasks in the job queue to complete, before returning to the for-in loop to receive completed futures from as_completed() and to print out the results. Notice that the results printed after the ValueError exception are printed at approximately the same time.

Meanwhile, when wait = False, Python will continue with the main thread’s execution flow.

2021-07-10 11:46:00.282762 - I'm running in a thread: 1
2021-07-10 11:46:00.283760 - I'm running in a thread: 2
2021-07-10 11:46:01.283596 - I'm running in a thread: 4
2021-07-10 11:46:01.284606 - EXCEPTION! Num is 5
2021-07-10 11:46:01.286596 - I'm running in a thread: 3
2021-07-10 11:46:02.285767 - I'm running in a thread: 7
2021-07-10 11:46:02.287289 - I'm running in a thread: 6
2021-07-10 11:46:03.296363 - I'm running in a thread: 9
2021-07-10 11:46:03.297305 - I'm running in a thread: 8
2021-07-10 11:46:03.299306 - Run complete

Since wait = False, after handling the ValueError exception, the main thread immediately continues with the script’s execution flow, which is to wait and receive completed futures from as_completed() and to print out the results. Notice that the results printed after the ValueError exception are printed as each future completes.

You can check out the shutdown() code for Python 3.7 and Python 3.8 at GitHub.

Python 3.9

At the time of writing this blog post, I was using Python 3.9.5.

In Python 3.9, an additional boolean parameter, cancel_futures, was added to shutdown(). This allows us to cancel pending futures in the queue that have not started running if any future fails mid-way through. However, there is a catch. If you use the as_completed() function, the script will be stuck indefinitely after calling executor.shutdown() because as_completed() is not triggered when a future is in the CANCELLED state.

As you can see here, as_completed() only returns a future if it is in the CANCELLED_AND_NOTIFIED or FINISHED state. Meanwhile, executor.shutdown() only cancels the future but does not change its state to CANCELLED_AND_NOTIFIED.

To confirm it, we can run this block of code below.

from concurrent.futures import ThreadPoolExecutor, as_completed
from datetime import datetime
import time

def task(num):
    if num == 5:
        raise ValueError("Num is 5")
    else:
        # To ensure this task doesn't complete too fast
        time.sleep(1)
    return f"I'm running in a thread: {num}"

with ThreadPoolExecutor(2) as pool:
    futures = [
        pool.submit(task, i) for i in range(1,10)
    ]

    for future in as_completed(futures):
        print([f._state for f in futures])
        try:
            n = future.result()
            print(f"{datetime.now()} - {n}")
        except ValueError as e:
            print(f"{datetime.now()} - EXCEPTION! {e}")
            pool.shutdown(wait=False, cancel_futures=True)

print(f"{datetime.now()} - Run complete")

We are printing the states of all the futures every time we enter the for-in loop. The results of the block of code above is as shown below:

['RUNNING', 'FINISHED', 'RUNNING', 'PENDING', 'PENDING', 'PENDING', 'PENDING', 'PENDING', 'PENDING']
2021-07-10 12:24:10.208194 - I'm running in a thread: 2
['FINISHED', 'FINISHED', 'RUNNING', 'RUNNING', 'PENDING', 'PENDING', 'PENDING', 'PENDING', 'PENDING']
2021-07-10 12:24:10.209194 - I'm running in a thread: 1
['FINISHED', 'FINISHED', 'FINISHED', 'FINISHED', 'FINISHED', 'RUNNING', 'RUNNING', 'PENDING', 'PENDING']
2021-07-10 12:24:11.212126 - I'm running in a thread: 3
['FINISHED', 'FINISHED', 'FINISHED', 'FINISHED', 'FINISHED', 'RUNNING', 'RUNNING', 'PENDING', 'PENDING']
2021-07-10 12:24:11.213099 - EXCEPTION! Num is 5
['FINISHED', 'FINISHED', 'FINISHED', 'FINISHED', 'FINISHED', 'RUNNING', 'RUNNING', 'CANCELLED', 'CANCELLED']
2021-07-10 12:24:11.215099 - I'm running in a thread: 4
['FINISHED', 'FINISHED', 'FINISHED', 'FINISHED', 'FINISHED', 'RUNNING', 'FINISHED', 'CANCELLED', 'CANCELLED']
2021-07-10 12:24:12.217437 - I'm running in a thread: 7
['FINISHED', 'FINISHED', 'FINISHED', 'FINISHED', 'FINISHED', 'FINISHED', 'FINISHED', 'CANCELLED', 'CANCELLED']
2021-07-10 12:24:12.218478 - I'm running in a thread: 6

We can see that after the ValueError exception is handled, the last two futures changes state from PENDING to CANCELLED. The script will then be stuck because as_completed() never returns the remaining two futures.

What To Do

The way I fixed this is to scrap using the as_completed() function. Although it is the recommended way to loop through a list of futures, it’s useless if it causes the script to hang.

from concurrent.futures import ThreadPoolExecutor, as_completed
from datetime import datetime
import time

def task(num):
    if num == 5:
        raise ValueError("Num is 5")
    else:
        # To ensure this task doesn't complete too fast
        time.sleep(1)
    return f"I'm running in a thread: {num}"

with ThreadPoolExecutor(2) as pool:
    futures = [
        pool.submit(task, i) for i in range(1,10)
    ]

    for future in futures:
        print([f._state for f in futures])
        if future.cancelled():
            continue
        try:
            n = future.result()
            print(f"{datetime.now()} - {n}")
        except ValueError as e:
            print(f"{datetime.now()} - EXCEPTION! {e}")
            pool.shutdown(wait=False, cancel_futures=True)

print(f"{datetime.now()} - Run complete")

Now, we are going through the list of futures sequentially. We are also checking if the future is in CANCELLED state and if it is, we proceed to the next future. We have successfully avoided the never-ending as_completed() function when .shutdown(cancel_futures=True) is called.

The output of the code above will look like:

['RUNNING', 'RUNNING', 'PENDING', 'PENDING', 'PENDING', 'PENDING', 'PENDING', 'PENDING', 'PENDING']
2021-07-10 12:46:30.983909 - I'm running in a thread: 1
['FINISHED', 'FINISHED', 'RUNNING', 'RUNNING', 'PENDING', 'PENDING', 'PENDING', 'PENDING', 'PENDING']
2021-07-10 12:46:30.986902 - I'm running in a thread: 2
['FINISHED', 'FINISHED', 'RUNNING', 'RUNNING', 'PENDING', 'PENDING', 'PENDING', 'PENDING', 'PENDING']
2021-07-10 12:46:32.002421 - I'm running in a thread: 3
['FINISHED', 'FINISHED', 'FINISHED', 'FINISHED', 'FINISHED', 'RUNNING', 'RUNNING', 'PENDING', 'PENDING']
2021-07-10 12:46:32.004655 - I'm running in a thread: 4
['FINISHED', 'FINISHED', 'FINISHED', 'FINISHED', 'FINISHED', 'RUNNING', 'RUNNING', 'PENDING', 'PENDING']
2021-07-10 12:46:32.006840 - EXCEPTION! Num is 5
['FINISHED', 'FINISHED', 'FINISHED', 'FINISHED', 'FINISHED', 'RUNNING', 'RUNNING', 'CANCELLED', 'CANCELLED']
2021-07-10 12:46:33.008187 - I'm running in a thread: 6
['FINISHED', 'FINISHED', 'FINISHED', 'FINISHED', 'FINISHED', 'FINISHED', 'FINISHED', 'CANCELLED', 'CANCELLED']
2021-07-10 12:46:33.010195 - I'm running in a thread: 7
['FINISHED', 'FINISHED', 'FINISHED', 'FINISHED', 'FINISHED', 'FINISHED', 'FINISHED', 'CANCELLED', 'CANCELLED']
['FINISHED', 'FINISHED', 'FINISHED', 'FINISHED', 'FINISHED', 'FINISHED', 'FINISHED', 'CANCELLED', 'CANCELLED']
2021-07-10 12:46:33.013183 - Run complete

Bonus: I Want To Cancel Futures On Shutdown in Python 3.7 and 3.8

With a bit of knowledge of object-oriented programming, we can! Simply inherit the ThreadPoolExecutor class and add a new method (in case we still want the original shutdown()) with Python 3.9’s version of shutdown().

from concurrent.futures import ThreadPoolExecutor
import queue

class CustomThreadPoolExecutor(ThreadPoolExecutor):
    def shutdown39(self, wait=True, *, cancel_futures=False):
        with self._shutdown_lock:
            self._shutdown = True
            if cancel_futures:
                # Drain all work items from the queue, and then cancel their
                # associated futures.
                while True:
                    try:
                        work_item = self._work_queue.get_nowait()
                    except queue.Empty:
                        break
                    if work_item is not None:
                        work_item.future.cancel()

            # Send a wake-up to prevent threads calling
            # _work_queue.get(block=True) from permanently blocking.
            self._work_queue.put(None)
        if wait:
            for t in self._threads:
                t.join()

Populating MySQL Docker Image

A tutorial on starting up a MySQL Docker image on your local machine populated with some dummy data for local testing

December 11, 2020 1 min read

I had the opportunity to work on more programming tasks in the past year and had to test my code which has to connect to MySQL databases. Of course, you can write mocks for unit testing, but I also wanted to do a little integration testing to actually connect to the database and retrieve some data on my local machine. Therefore, I decided to prepare a GitHub repository which allows me to easily spin up a MySQL Docker container using Docker Compose, which is populated with dummy data.

You can find the GitHub repository here.

To put it simply, assuming you have Git and Docker installed, you just need to run:

git clone https://github.com/TiewKH/mysql-docker-dummy
cd ./mysql-docker-dummy
docker-compose up

If the dummy data does not suffice, you just have to add new SQL scripts into the sql_scripts directory and when you start up the container with Docker Compose, the MySQL database will be populated with your data.

Tensorflow 2.0 / Keras - LSTM vs GRU Hidden States

Exploring cell state and hidden state for LSTM and GRU while doing Tensorflow's Neural Machine Translation with Attention tutorial.

June 25, 2019 4 min read

I was going through the Neural Machine Translation with Attention tutorial for Tensorflow 2.0. Having gone through the verbal and visual explanations by Jalammar and also a plethora of other sites, I decided it was time to get my hands dirty with actual Tensorflow code.

I had previously done a bit of coding related to CNNs for my Final Year Project, but this was my first experience with RNNs. I had learnt the theory of RNNs and its mainstream relatives, LSTM and GRU, but this was my first time looking at the code. The first thing that got me stumped was the hidden states in the Encoder class.

class Encoder(tf.keras.Model):
  def __init__(self, vocab_size, embedding_dim, enc_units, batch_sz):
    super(Encoder, self).__init__()
    self.batch_sz = batch_sz
    self.enc_units = enc_units
    self.embedding = tf.keras.layers.Embedding(vocab_size, embedding_dim)
    self.gru = tf.keras.layers.GRU(self.enc_units,
                                   return_sequences=True,
                                   return_state=True,
                                   recurrent_initializer='glorot_uniform')

  def call(self, x, hidden):
    x = self.embedding(x)
    output, state = self.gru(x, initial_state = hidden)
    return output, state

  def initialize_hidden_state(self):
    return tf.zeros((self.batch_sz, self.enc_units))

The key here are the two arguments, return_sequences and return_state, which were both set to True. This will enable us to retrieve the output state and hidden state. Initially, I thought that the output state and the hidden state were the same. This was because my knowledge of LSTM and GRU were lacking.

Fig 1. - LSTM and GRU image taken from TowardsDataScience

As we can see, LSTM has two outputs, the cell state on top, and the hidden state at the bottom. Meanwhile, GRU only has one output, the hidden state. After going through Keras’ documentation (because Tensorflow 2.0’s documentation has not been fully updated), it is stated that:

Output shape

if return_state: a list of tensors. The first tensor is the output. The remaining tensors are the last states, each with shape (batch_size, units). For example, the number of state tensors is 1 (for RNN and GRU) or 2 (for LSTM).
if return_sequences: 3D tensor with shape (batch_size, timesteps, units).

A minimal example is available at Understand the Difference Between Return Sequences and Return States for LSTMs in Keras by Jason Brownlee. The code below is extracted from the post linked above and is for LSTM.

from keras.models import Model
from keras.layers import Input
from keras.layers import LSTM
from numpy import array
inputs1 = Input(shape=(3, 1))
lstm1, state_h, state_c = LSTM(1, return_sequences=True, return_state=True)(inputs1)
model = Model(inputs=inputs1, outputs=[lstm1, state_h, state_c])
data = array([0.1, 0.2, 0.3]).reshape((1,3,1))
print(model.predict(data))

The output for that piece of code would be:

[array([[[-0.02145359],
        [-0.0540871 ],
        [-0.09228823]]], dtype=float32),
 array([[-0.09228823]], dtype=float32),
 array([[-0.19803026]], dtype=float32)]

However, when we use GRU, keep in mind that it has no cell state. It only has a hidden state.

from keras.models import Model
from keras.layers import Input
from keras.layers import GRU
from numpy import array
inputs1 = Input(shape=(3, 1))
output, hidden = GRU(1, return_sequences=True, return_state=True)(inputs1)
model = Model(inputs=inputs1, outputs=[output, hidden])
data = array([0.1, 0.2, 0.3]).reshape((1,3,1))
print(model.predict(data))

The output for that piece of code would be:

[array([[[-0.02145359],
        [-0.0540871 ],
        [-0.09228823]]], dtype=float32),
 array([[-0.09228823]], dtype=float32)]

Notice that the second array is the final hidden state, which is the same as the last value of the first array. The output from the GRU also has one less array because it does not have a cell state unlike LSTM.

Now, pardon me as I get back to completing the tutorial.

Editing Newest JSON File In A Directory

This post shows how to edit the newest (by create date) JSON file from a Windows directory

June 21, 2019 2 min read

This post shows how to edit the newest (by create date) JSON file from a Windows directory. In this example, we are using Python 3.6.

First, we have to load the needed libraries.

from stat import S_ISREG, ST_CTIME, ST_MODE
import os, time
import json

Then, we have to get all JSON files from a directory. Replace the directory here with the directory you want.

directory = "C:/Users/tiewkeehui/JsonFiles/"
entries = (os.path.join(directory, fn) for fn in os.listdir(directory) if fn.endswith('.json'))
entries = ((os.stat(path), path) for path in entries)

Then we add the stats of all the files

entries = ((stat[ST_CTIME], path) for stat, path in entries if S_ISREG(stat[ST_MODE]))

S_ISREG checks if the file is a normal file (ex. not a directory). It will return a non-zero value if it is a regular file.

Now we get the latest created JSON file

json_file_path = sorted(entries)[-1][1]

Last but not least, we open and load the JSON file and add our new key-value pair. After that, we overwrite the existing file.

with open(json_file_path, encoding="utf-8-sig") as json_file:
    data = json.load(json_file)
    data['new_key'] = new_value

    with open(json_file_path, 'w') as f:
        json.dump(data, f)

References:

Association Rule Mining on the Extended Bakery dataset

January 30, 2017 5 min read

Introduction

We used the Extended Bakery Dataset’s 75,000 receipt data from apriori.zip, which can be found at this website.

We also used EB-build-goods.sql to convert the product IDs to their names. The original file can be found here.

Objective

What is the domain and what are the potential benefits to be derived from association rule mining?

The domain of association rule mining is that it is a mining method that specialises in finding frequent patterns, associations, correlations or causal structures in the ExtendedBakery data set that is provided. With associative rule mining we can possible get to improve the inventory management, customer buying prediction and time related sales.

Association rules mining or what is sometimes referred to as ‘Market Basket Analysis’ is among the preeminent component used in data mining to find useful insights to a particular domain. It is a rule-based machine learning method designed to discover frequent co-occurring associations among a collection of items in transaction and even in relational databases. Normally, data produced in transactions are categorical(non-numeric) data which makes association rules mining a pertinent method because it handles these forms of data well when searching for interesting discoveries.

An association rule has two parts, an antecedent (if) and a consequent (then). An antecedent is an item found in the data. A consequent is an item that is found in combination with the antecedent. The strength of an association rule can be measured in terms of its support and confidence. Support is an indication of how frequently the items appear in the database. This is of interest due to the fact that if a rule is measured to be very low in support, it is likely to be uninteresting from a business perspective. For example, it may prove unprofitable to promote items that customers seldom buy together. Confidence, on the other hand indicates the number of times the if/then statements have been found to be true. It essentially measures the reliability of the inference made by a rule.

The classic example of the Beer and Diapers association that is often mentioned in data mining books. The association suggests that a strong relationship exists between the sale of diapers and beer because many customers who buy diapers also buy beer. This can help retailers to learn about purchasing behavior of their customers. Such information can also be utilized to support a variety of business-related applications such as marketing promotions, inventory management, and customer relationship management.

Association mining has been broadly used in many application domains besides the business field in the last years the application areas have increased significantly. Some recent applications are the discovery of patterns in biological databases, extraction of knowledge from software engineering metrics and the optimization of user’s profiles for web system personalization. An example would be a case for Walmart in 2004 when a series of hurricanes crossed the state of Florida. Walmart mined their massive retail transaction database to see what their customers really wanted to buy prior to the arrival of a hurricane. They found one particular item that increased in sales by a factor of 7 over normal shopping days. That was a huge Lift factor for a real-world case. That one item was not bottled water, or batteries, or beer, or flashlights, or generators, or any of the usual things that we might imagine. The item was strawberry pop tarts. Therefore, Walmart stocked their stores with tons of strawberry pop tarts prior to the next hurricanes, and they sold them out. That is a win-win: Walmart wins by making the sell, and customers win by getting the product that they most want.

Now that it has been established what association rules are and how it is utilised, we are able to continue on how would one would be able to apply this to the current Extended Bakery dataset. The question now comes to, what good may come from the relationships and rules that will be found? Overall, many growth inducing outcomes can occur from the gleaning and utilization of rules. First of all, this would result in the owners of the business being much more knowledgeable on the subject of their own business. After pondering this information, one can also form many courses of action based on solid theories backed by proof rather than on simple guesses. The business may increase its sales of specific items of their shop by displaying items with higher correlation together. This should boost the amount of sales of this set, as it increases awareness of the presence of the other item/s in the set. They can also choose sets of items that sell well together to offer as a much more desirable form of promotion, for example set meals and coupon discounts. This option is viable as it would draw more customers into the shop, and thus increase sales and awareness of the shop. In conclusion, using the association rules may result in higher sales and also increases awareness of the store, both of the potential customers and the owner’s.

Legacy interactive/output embed. Open full page

The R Script for the exploratory exercise can be found here.

Exploratory Data Analysis on the Student Alcohol Consumption dataset (Introduction)

December 30, 2016 13 min read

The dataset we chose is the Student Alcohol Consumption dataset by UCI Machine Learning which can be obtained at Kaggle. This analysis was done as part of fulfilling the Data Mining course in Multimedia University. People who contributed to this were Aaron Patrick Nathaniel, Lim Yue Hng (Neil) and Nicolas Raj.

Describing the dataset

Alcohol is an often abused substance that troubles many individuals in their adulthood as they struggle to cope with emotional and physical stress that comes with the mantle of adulthood. However, many do not consider the effect of this intoxicating substance in the context of the younger, more impressionable generation. Therefore, researchers seek to rectify that lack by conducting a survey to obtain important raw data on alcohol consumption in a student environment as well as their demographic information and other data that may be of some relevance.

This Student Alcohol Consumption dataset is based on data collected in two secondary schools in Portugal. The students included in the survey were in the courses of mathematics and Portuguese. The amount of mathematics students involved in the collection was 395, whereas 649 Portuguese Language students were recorded to have participated. However, the data reveals that there was a total of 382 students that were in both datasets, this was evident in the exact replication of data. The data collected, in locations such as Gabriel Pereira and Mousinho da Silveira, includes several values of pertinence. Examples of such data are records of demographic information, grades, and alcohol consumption. The dataset which we will be exploring will be the dataset containing information about the students from the mathematics course only.

The datasets have a total of 33 attribute columns of which we could do some column selection based on certain parameters. This will be attempted in section E as part of the preprocessing before plotting the data for our exploratory data analysis. The columns and how they are recorded are as listed below:

school - GP/MS for the student's school
sex - M/F for gender
age - 15-22 for the student's age
address - U/R for urban or rural respectively
famsize - LE3/GT3 for less than or greater than three family members
Pstatus - T/A for living together or apart from parents, respectively
Medu - 0 (none) / 1 (primary-4th grade) / 2 (5th - 9th grade) / 3 (secondary) / 4 (higher) for mother's education
Fedu - 0 (none) / 1 (primary-4th grade) / 2 (5th - 9th grade) / 3 (secondary) / 4 (higher) for father's education
Mjob - 'teacher', 'health' care related, civil 'services', 'at_home' or 'other' for the student's mother's job
Fjob - 'teacher', 'health' care related, civil 'services', 'at_home' or 'other' for the student's father's job
reason - close to 'home', school 'reputation', 'course' preference or 'other' for the choice of school
guardian - mother/father/other as the student's guardian
traveltime - 1 (<15mins) / 2( 15 - 30 mins) / 3 (30 mins - 1 hr) / 4 (>1hr) for time from home to school
studytime - 1 (<2hrs) / 2 (2 - 5hrs) / 3 (5 - 10hrs) / 4 (>10hrs) for weekly study time
failures - 1-3/4 for number of class failures (if more than 3 than record 4)
schoolsup - yes/no for extra educational support
famsup - yes/no for family educational support
paid - yes/no for extra paid classes for Math or Portuguese
activities - yes/no for extra-curricular activities
nursery - yes/no for whether attended nursery school
higher - yes/no for desire to continue studies
internet - yes/no for internet access at home
romantic - yes/no for relationship status
famrel - 1-5 scale on quality of family relationships
freetime - 1-5 scale on how much free time after school
goout - 1-5 scale on how much student goes out with friends
Dalc - 1-5 scale on how much alcohol consumed on weekdays
Walc - 1-5 scale on how much alcohol consumed on weekend
health - 1-5 scale on health condition
absences - 0-93 amount of absences from school
G1 - 0-20 for the first period grade
G2 - 0-20 for the second period grade
G3 - 0-20 for the final grade

Possible insights that can be obtained from mining the chosen dataset

Since the main purpose of the dataset is to find correlations between students with their alcohol consumption patterns, the most conspicuous relationship would be the relationship between their grades with respect to their workday and weekend alcohol consumption. The traditional consensus is that students who consume alcohol at high levels tend to skip more classes and perform worse in their studies, thus, resulting in lower grades. However, a research conducted in the United States by Balsa (2011), showed that increases in levels of alcohol consumption only resulted in small reductions of GPA. We shall see which consensus holds true. Five columns play a major role in this which are: column 27 (workday alcohol consumption) and/or column 28 (weekend alcohol consumption), column 31 (first period grade), column 32 (second period grade) and column 33 (final grade).

For the data exploratory exercise, we choose to examine four columns: workday alcohol consumption, first period grade, second period grade and their final grade. We chose workday alcohol consumption because drinking over workdays is more unusual than drinking over the weekends. Our explanation would be more focused on the final grade because we think that students will be more serious towards their final grade rather than the first period grade and second period grade. Thus, their final grade would be the perfect measure of that particular student's success. However, if more elaborate data mining techniques were to be used, more features can be selected and used in order to obtain more accurate insights. This will be explained in the next section (Section C).

As we all know, human relationships play a major role in people's lives. It can develop a plethora of emotions in oneself, may it be a positive or negative emotion. It is a usual train of thought that those who have a bad relationship with their family members will be stressed and unhappy which results in them drinking alcohol for consolation. We could check to see if that hypothesis has a concrete basis by using column 24 (famrel), column 27 (workday alcohol consumption) and/or column 28 (weekend alcohol consumption). If the hypothesis holds true, we would expect to see an increasing level of alcohol consumption (both column 27 and 28) when famrel has a low value. However, the assumption is that the alcohol consumption is high because the student's relationship with his/her family has a low value. Correlation does not imply causation. This may not hold true because it is a possibility that the student's relationship with his/her family is low because of their high level of alcohol consumption. For the data exploratory exercise, we choose to examine three columns: workday alcohol consumption, weekend alcohol consumption and their family relationship.

Besides family relationships, we can also try to find if there is a relationship between being single and consuming high levels of alcohol. To do so, we need to take column 23 (romantic), column 27 (workday alcohol consumption) and/or column 28 (weekend alcohol consumption) into consideration. Column 23 (romantic), only gives information on whether or not the student has a partner. It does not state the level of intimacy between them. A research conducted by Dinescu et. al. (2016), studied the relationship between married couples with their single counterparts and found out that if partners are more intimate, they will drink less. Assuming the romantic relationship in our dataset is of an intimate level, we can find out if this statement holds true. For the data exploratory exercise, we choose to examine three columns: workday alcohol consumption, weekend alcohol consumption and their relationship status.

Last but not least, we can also obtain insights on health issues and drinking alcohol. In general, we would assume that people who are not healthy, will avoid drinking in order to prevent their health from further deterioration. To obtain insights on this, we could refer to column 29 (health), column 27 (workday alcohol consumption) and/or column 28 (weekend alcohol consumption). We would think that if the value for health is lower, the value for their workday and/or weekend alcohol consumption would also be lower. For the data exploratory exercise, we choose to examine three columns: workday alcohol consumption, weekend alcohol consumption and their health.

What type of data mining technique would be relevant?

The data mining technique we think is suitable is classification. For a student to pass the subject, there are a couple of factors that could be correlated with the outcome. We could take into consideration the following:

the amount of time a student studies (studytime, column 14)
does the student join any extra paid classes (paid, column 18)
does the student participate in any extra co-curricular activities (activities, column 19)
if the student is involved in any romantic relationship (romantic, column 23)
how is the student's family relationship quality (famrel, column 24)
the tendency of the student to go out with friends (goout, column 26)
weekday alcohol consumption (Dalc, column 27)
weekend alcohol consumption (Walc, column 28)

Figure 1 illustrates the high-level description of our classification.

Figure 1: Using GStatus as the class

We think that classification is the best data mining technique to be employed because we can build a classification model to predict if a student will get a passing grade based on the factors mentioned above. We can use studytime (column 14), paid (column 18), activites (column 19), romantic (column 23), famrel (column 24), goout (column 26), Dalc (column 27), Walc (column 28) as the attributes and GStatus as the class for the training set to predict the class GStatus in the test set and validate the model.

GStatus is derived from the final period grade, (G3, column 33) where according to EuroEducation.net (n.d.), the passing marks for a student in Portugal would be 10 out of 20. The reason for this change is because it is easier to classify a student's result as pass/fail rather than a discrete numeric number. This would help the classification model to more accurately predict the class GStatus because it would be less accurate for the classification model to predict a numeric value ranging from 0-20.

Data quality issues

There are a few columns which we think could be further clarified or changed. They are:

guardian (column 12) - Currently, it is divided into 3 levels which are "mother", "father" and "other". However the part labelled "other" should be a bit more specific as for the entries where people have placed "other". They should add factors of whether the student is an orphan or living with their closer relatives.

	<li><b>schoolsup (column 16)</b> - The column name schoolsup with entries "yes" or "no" of which stands for extra educational support is a
	little misleading and ambiguous as we do not know if it means the student is receiving extra educational support due to this grades/results or because
	the student is from a unwealthy family and is receiving support in terms of food. If it is the latter, it will not have a big impact on grades compared
	to the former. Instead, schoolsup should be replaced with a column named scholarship to indicate whether if a student is under the school
	scholarship. The value for the attribute should be a simple binary answer such as yes/no.</li>
		
	<li><b>paid (column 18)</b> - This column is for students with extra paid classes within the course subject which only has yes or no option, it does 
	not state how many paid classes they took. A proper amendment would be to ask how many extra paid classes they took in a week and have the values 
	be of an numeric format.</li>
		
	<li><b>activities (column 19)</b> - This column is for extra-curricular activities that students have participated in and it only has a yes or no option.
	There is no indication as to how many extra-curricular the students have taken. A proper way to go about this is to have the column ask how many 
	extra-curricular activities they took in a week and have the values be of an numeric format.</li>
		
	<li><b>goout (column 26)</b> - This column is for the amount of time that the student spend to go out with his/her friends. The option given is to state how much 
	they go out with friends (1 if very low and 5 if very high). This is not a proper measurement unit. It should have been recorded as how many times the student
	go out with their friends in a week and the values should be in a numeric format. </li>

References

Balsa, A. I., Giuliano, L. M., & French, M. T. (2011). The effects of alcohol use on academic achievement in high school. Economics of Education Review, 30(1), 1-15.
Dinescu, D., Turkheimer E., Beam, C.R., Horn, E.E., Duncan, G., Emery, R.E. (2016). Is marriage a buzzkill? A twin study of marital status and alcohol consumption. Journal of Family Psychology, Vol 30(6), Sep 2016, 698-707.
EuroEducation.net. (n.d.). EDUCATION SYSTEM IN PORTUGAL. Retrieved from http://www.euroeducation.net/prof/porco.htm