🌟 Hit star button to save this repo in your profile

About Us

The information on this Github is part of the materials for the subject High Performance Data Processing (SECP3133). This folder contains general Exploratory Data Analysis (EDA) information as well as EDA case studies using Malaysian datasets. This case study was created by a Bachelor of Computer Science (Data Engineering), Universiti Teknologi Malaysia student.

Exploratory Data Analysis

Exploratory Data Analysis (EDA) is a crucial step in the data analysis process that involves examining and summarizing a dataset to understand its characteristics, identify patterns, and gain insights into the data. EDA is typically performed before more advanced statistical and machine learning techniques are applied and helps in forming hypotheses, selecting appropriate modeling approaches, and ensuring data quality. Here are some key components and techniques used in EDA:

Exploratory Data Analysis (EDA) is a crucial step in the data analysis process that involves examining and summarizing a dataset to understand its characteristics, identify patterns, and gain insights into the data. EDA is typically performed before more advanced statistical and machine learning techniques are applied and helps in forming hypotheses, selecting appropriate modeling approaches, and ensuring data quality. Here are some key components and techniques used in EDA:

1. Data Summary: Begin by understanding the basic information about the dataset, such as the number of rows and columns, data types, missing values, and summary statistics (mean, median, standard deviation, etc.).

2. Data Visualization: Visualizing data through plots and charts can provide a clearer understanding of its distribution and patterns. Common types of visualizations include histograms, box plots, scatter plots, and bar charts.

3. Data Distribution: Analyze the distribution of variables to determine whether they follow normal, uniform, or other types of distributions. This can impact the choice of statistical tests and modeling techniques.

4. Correlation Analysis: Explore the relationships between variables using correlation matrices, scatter plots, and other correlation measures. This helps identify potential dependencies and multicollinearity.

5. Outlier Detection: Identify and handle outliers in the data. Outliers can significantly affect statistical measures and model performance.

6. Categorical Variables: Examine the distribution of categorical variables through frequency tables, bar plots, and pie charts. This helps understand the composition of categorical data.

7. Data Transformation: Apply transformations (e.g., log transformation, standardization) to make the data more suitable for analysis, especially if it doesn't meet assumptions of statistical methods.

8. Feature Engineering: Create new variables or features that might be more informative or relevant for the analysis. This could involve aggregating, combining, or extracting information from existing variables.

9. Missing Data Handling: Deal with missing data, either by imputing missing values or excluding incomplete records. The choice of method depends on the nature of the data and the problem at hand.

10. Hypothesis Testing: If relevant, perform hypothesis tests to determine whether observed differences or relationships in the data are statistically significant.

11. Data Transformation: Consider scaling or encoding categorical variables for modeling. This can include one-hot encoding, label encoding, or other techniques.

12. Dimensionality Reduction: Use techniques like Principal Component Analysis (PCA) or t-Distributed Stochastic Neighbor Embedding (t-SNE) to reduce the dimensionality of the data while preserving important information.

13. Time Series Analysis: For time series data, analyze trends, seasonality, and autocorrelation patterns. Techniques like autocorrelation plots and decomposition can be helpful.

14. Geospatial Analysis: When dealing with geographic data, use maps, geospatial plots, and spatial statistics to understand spatial patterns and relationships.

15. Text Analysis: If the dataset contains text data, perform text mining and sentiment analysis to extract insights from the textual content.

EDA is an iterative process, and the specific techniques and tools used can vary depending on the nature of the data and the objectives of the analysis. It plays a crucial role in gaining an initial understanding of the data, guiding subsequent analysis, and making informed decisions about the next steps in a data science or analytical project.

Why is EDA so important in data science?

✅️ The main purpose of EDA is to help you look at the data before making any assumptions. In addition to better understanding the patterns in the data or detecting unusual events, it also helps you find interesting relationships between variables.

✅️ Data scientists can use exploratory analysis to ensure that the results they produce are valid and relevant to desired business outcomes and goals.

✅️ EDA also helps stakeholders by verifying that they are asking the right questions.

✅️ EDA can help to answer questions about standard deviations, categorical variables, and confidence intervals.

✅️ After the exploratory analysis is completed and the predictions are determined, its features can be used for more complex data analysis or modeling, including machine learning.

Python

👉 Python is a popular programming language for data science and has several libraries and tools that are commonly used for EDA such as:

1. Pandas: a library for data manipulation and analysis.
2. Numpy: a library for numerical computing in Python.
3. Scikit-learn: Scikit-learn is a machine learning library, but it also includes tools for data preprocessing, feature selection, and dimensionality reduction, which are essential for EDA.
4. Matplotlib: a plotting library for creating visualizations.
5. Seaborn: a library based on matplotlib for creating visualizations with a higher-level interface.
6. Plotly: an interactive data visualization library.

In EDA, you might perform tasks such as cleaning the data, handling missing values, transforming variables, generating summary statistics, creating visualizations (e.g. histograms, scatter plots, box plots), and identifying outliers. All of these tasks can be done using the above libraries in Python.

📖 Notes

Basic Concept

• developers.google: Good Data Analysis
• Datascience using Python: Exploratory_Data_Analysis
• Towardsdatascience: What is Exploratory Data Analysis?
• Wikipedia: Exploratory data analysis
• r4ds: Exploratory Data Analysis
• careerfoundry:What Is Exploratory Data Analysis?
• How To Conduct Exploratory Data Analysis in 6 Steps
• A Five-Step Guide for Conducting Exploratory Data Analysis
• I asked ChatGPT to do Exploratory Data Analysis with Visualizations

Code & Practice

• simplilearn: What is Exploratory Data Analysis? Steps and Market Analysis
• Exploratory Data Analysis (EDA): Types, Tools, Process
• projectpro: Exploratory Data Analysis in Python-Stop, Drop and Explore
• medium.com: 10 Things to do when conducting your Exploratory Data Analysis (EDA)
• towardsdatascience.com: An Extensive Step by Step Guide to Exploratory Data Analysis
• EDA - Exploratory Data Analysis: Using Python Functions
• Step-by-Step Exploratory Data Analysis (EDA) using Python

Videos

• Exploratory Data Analysis Tutorial | What Is EDA | How EDA Works | EDA In Python | Intellipaat
• Live Day 1-Live Session On EDA And Feature Engineering- Zomato Dataset
• Live Day 2-Live Session On EDA And Feature Engineering- Black Friday Dataset
• Live Day 3-Live Session On EDA And Feature Engineering- Flight Price Prediction Dataset
• Step By Step Process In EDA And Feature Engineering In Data Science Projects
• Exploratory Data Analysis(EDA) of Titanic dataset
• Exploratory Data Analysis (EDA) Using Python | Python Data Analysis | Python Training | Edureka
• Exploratory Data Analysis with Pandas Python
• How to Do Data Exploration (step-by-step tutorial on real-life dataset)

Kaggle: Notebook

• Exploratory Data Analysis (Step by Step)
• A Simple Tutorial on Exploratory Data Analysis
• Intro to Exploratory data analysis (EDA) in Python
• Topic 1. Exploratory Data Analysis with Pandas
• Detailed exploratory data analysis with python
• EDA using Python Pandas
• Pandas: EDA of Cars Dataset
• Step-by-step Data Preprocessing & EDA

Github

📖 Lab

No	Dataset	Colab	GitHub
1	Boston
2	Car Features and MSRP
3	Housing Dataset
4	United Nations Development Corporation

🌟 Case Study: Exploratory Data Analysis

The provided list comprises a collection of case studies, each with a title and accessibility information on platforms like Colab and GitHub. These case studies likely involve data analysis and exploration. For instance, "404 Error" may involve exploring property-related data in Kuala Lumpur, while "Alrite" could be centered around the exportation of plantation products in Sarawak. "BEFE" appears to focus on COVID-19 clusters in Malaysia, "Boboiboy" on property listings in Kuala Lumpur, and "COLBY" on the results of the 14th General Election in Malaysia. "FANTOM" likely tracks daily recorded COVID-19 cases at the state level, "HAHA" pertains to foreign direct investment in Malaysia, and "HD" may involve land usage analysis in Tampin for 2021. Other case studies cover topics such as elections, healthcare, real estate, population, and more, providing a diverse range of data exploration possibilities.

Automated EDA Tools

EDA is a vital but time-consuming task in a data project. Here are 10 open-source tools that generate an EDA report in seconds.

Library	Description	Web	Github
SweetViz	- In-depth EDA report in two lines of code. - Covers information about missing values, data statistics, etc. - Creates a variety of data visualizations. - Integrates with Jupyter Notebook.	🌐
Pandas-Profiling	- Generate a high-level EDA report of your data in no time. - Covers info about missing values, data statistics, correlation etc. - Produces data alerts. - Plots data feature interactions.	🌐
DataPrep	- Supports Pandas and Dask DataFrames. - Interactive Visualizations. - 10x Faster than Pandas based tools. - Covers info about missing values, data statistics, correlation etc. - Plots data feature interactions.	🌐
AutoViz	- Supports CSV, TXT, and JSON. - Interactive Bokeh charts. - Covers info about missing values, data statistics, correlation etc. - Presents data cleaning suggestions.	🌐
D-Tale	- Runs common Pandas operation with no-code. - Exports code of analysis. - Covers info about missing values, data statistics, correlation etc. - Highlights duplicates, outliers, etc. - Integrates with Jupyter Notebook.	🌐
dabl	- Primarily provides visualizations. - Covers wide range of plots: Scatter pair plots. Histograms. - Target distribution.	🌐
QuickDA	- Get overview report of dataset. - Covers info about missing values, data statistics, correlation etc. - Produces data alerts. - Plots data feature interactions.	🌐
Datatile	- Extends Pandas describe(). - Provides column stats: column type count, missing, column datatype. - Mostly statistical information.	🌐
Lux	- Provides visualization recommendations. - Supports EDA on a subset of columns. - Integrates with Jupyter Notebook. - Exports code of analysis.	🌐
ExploriPy	- Performs statistical testing. - Column type-wise distribution: Continuous, Categorical - Covers info about missing values, data statistics, correlation etc.	🌐

Big Data: The Vital Role of Exploratory Data Analysis (EDA)

Big Data refers to the vast and complex datasets that exceed the capabilities of traditional data processing tools. It is characterized by the three V's: volume (large amounts of data), velocity (rapid data generation), and variety (different data types). Exploratory Data Analysis (EDA), on the other hand, is a data analysis approach that involves summarizing, visualizing, and understanding the key characteristics of a dataset to uncover insights and patterns. In the context of Big Data, EDA plays a crucial role in making the data more manageable by identifying relevant subsets, trends, and anomalies, enabling data scientists to extract meaningful information and inform decision-making processes.

Feature Engineering

Feature engineering in data science is the process of selecting, transforming, and creating relevant attributes or variables from raw data to improve the performance of machine learning models. It involves identifying patterns, relationships, and meaningful information within the data, and then designing or modifying features to enhance the model's ability to make accurate predictions or classifications. Effective feature engineering can lead to increased model accuracy, reduced overfitting, and a better understanding of the underlying data, making it a critical step in the data preprocessing pipeline for machine learning tasks.

Contribution 🛠️

Please create an Issue for any improvements, suggestions or errors in the content.

You can also contact me using Linkedin for any other queries or feedback.