开源软件名称：

开源软件地址：

开源编程语言：

开源软件介绍：

Data Science Hacks, Tips and Tricks

Data Science Hacks is created and maintained by Analytics Vidhya for the data science community.

It includes a variety of tips, tricks and hacks related to data science, machine learning

These Hacks are for all the data scientists out there. It doesn’t matter if you are a beginner or an advanced professional, these hacks will definitely make you efficient!

Feel free to contribute your own data science hacks here. Make sure that your hack follows the contribution guidelines

This repository is part of the free course by Analytics Vidhya. To learn more of such awesome hacks visit Data Science Hacks, Tips and Tricks

• Data Science Hack #1 - Resource Downloader

How can you extract image data directly from chrome in one click? Imagine that you want to make your own machine learning project but you don't have enough data, it becomes a daunting task Worry not you can use the ResourceSaver extension to directly download data! Let's see how!

Steps:

1. Install the chrome extension from the given URL.
2. Go to Google Images or any webpage from where you want to save the data.
3. Open Inspect Element and click to ResourceSaver Tab
4. Click on the button Save All Resources and a zip file will be created.
5. Unzip the file and open folder encrypted-tbn0.gstatic.com
6. You can find the images here.

• Data Science Hack #2 Pandas Apply

Pandas Apply is one of the most commonly used functions for playing with data and creating new variables. It returns some value after passing each row/column of a data frame with some function. The function can be both default or user-defined.

• Data Science Hack #3 Pandas Boolean Indexing

It helps to select subset of data based on the value of the data in the dataframe

• Data Science Hack #4 Pandas Pivot Table

It is used to create MS Excel style spreadsheet. Levels in the pivot table will be stored in MultiIndex objects (hierarchical indexes) on the index and columns of the result DataFrame.

• Data Science Hack #5 Pandas crosstab

pd.crosstab() function is used to get an initial “feel” (view) of the data.

• Data Science Hack #6 Pandas str.split

It is used to apply vectorized string functions on a pandas dataframe column. Let’s say you want to split the names in a dataframe column into first name and last name. pandas.Series.str along with split( ) can be used to perform this task.

• Data Science Hack #7 Extract E-mails from text

Here is an interesting hack to extract email ids present in long pieces of text by just using 2 lines of code in Python using regular expressions. Extracting information from social media posts and websites has become a common practice in data analytics but sometimes we end up trying complicated methods to achieve things that can be solved easily by using the right technique.

• Data Science Hack #8 Normal Distribution

One of the most important assumptions in linear and logistic regression is that our data must follow a normal distribution but we all know that's usually not the case in real life. We often need to transform our data into normal/ gaussian distribution.

• Data Science Hack #9 Remove Emojis from text

Preprocessing is one of the key steps for improving the performance of a model. One of the main reasons for text preprocessing is to remove unwanted characters from text like punctuation, emojis, links and so on which is not required for our problem statement.

• Data Science Hack #10 Elbow method for classifier

Elbow Method is used for identifying the value of k in k-Nearest Neighbors. It's a plot of errors at different values of k and we select the k value having least error!

• Data Science Hack #11 MinMax Scaler

An important part of data analysis is to preprocessing. Many times we need to scale our features like in the case of k-NN we always need to scale the data before building model or else it'll give spurious results.

• Data Science Hack #12 Feature engineering for time series data

Most of the data collected today, hold the date and time variables. There is a lot of information that you can extract from these features and you can utilize it in your analysis!

• Data Science Hack #13 Dummy data for linear regression

Deeplearning models usually require a lot of #data for training. But acquiring massive amounts of data comes with its own challenges. Instead of spending days manually collecting data, you can make use of Image Augmentation techniques. It is the process of generating new images. These new images are generated using the existing training images and hence we don’t have to collect them manually.

• Data Science Hack #14 HuggingFace Tokenization

Tokenization is the primary task while building the vocabulary. HuggingFace recently created a library for tokenization which provides an implementation of today's most used tokenizers, with a focus on performance and versatility. Key features: Ultra-fast: They can encode 1GB of text in ~20sec on a standard server's CPU

• Data Science Hack #15 Divide Continuous and categorical data

You can extract categorical and numeric features into seperate dataframes in just 1 line of code! This can be done using the select_dtypes function.

• Data Science Hack #16 Pandas Profiling

Do you want to to do perform quick data analysis on your dataframe? You can use pandas profiling to generate profile report of your dataset in just 1 line of code!

• Data Science Hack #17 Formatting of DataFrame

Convert wide form dataframe into long form dataframe in just 1 line of code! In pd.melt(), one more columns are used as identifiers. "Unmelt the data", use pivot() function

• Data Science Hack #18 Magic Function- %history

Do you know how you can get the history of all the commands running inside your jupyter notebook? Use %history, jupyter notebook's built-in magic function! Note - Even if you have cut the cells in your notebook, %history will print those commands as well!

• Data Science Hack #19 Heatmap on pandas dataframe

Create heatmap on pandas dataframe using seaborn! It helps you understand the complete range of values at a glimpse.

• Data Science Hack #20 Plot confusion matrix

Scikit-learn has released its stable 0.22.1 version with new features and bug fixes. One new function is the plot_confusion_matrix function which generates an extremely intuitive and customisable confusion matrix for your classifier. Bonus tip: You can specify the format of the numbers appearing in the boxes using the values_format parameter('n' for whole numbers, '.2f' for float, etc)

• Data Science Hack #21 Ipython Interactive shell

What will be the output if you run the following commands in single cell of your jupyter notebook? df.shape df.head() Ofcourse it'll be first five rows of your dataframe. Can we get output of both the command run in same cell? You can do it using InteractiveShell.

• Data Science Hack #22 Python tqdm

Most of you have heard about the library tqdm and you might be using it track the progress of forever running for loops. Most of the times we write complex functions having nested for loops. #tqdm allows tracking that too. Here is how you can track the nested loops using tdqm in python.

• Data Science Hack #23 Image Augmentation

Deeplearning models usually require a lot of data for training. But acquiring massive amounts of data comes with its own challenges. Instead of spending days manually collecting data, you can make use of Image Augmentation techniques. It is the process of generating new images. These new images are generated using the existing training images and hence we don’t have to collect them manually.

• Data Science Hack #24 Setup Dark Jupyter Notebook Theme

jupyter-themes provides an easy way to change theme, fonts and much more in your jupyter notebook.

Steps -

1. Install jupyter-themes -
   • using anaconda

     conda install -c conda-forge jupyterthemes

   • using pip

     pip install jupyterthemes

2. Check list of themes -

   jt - l

3. Select a theme

   jt -t chesterish

4. To restore to default theme -

   jt -r

• Data Science Hack #25 Change cell width in jupyter notebook

To do this we use jupyter-themes, it provides an easy way to change theme, fonts and much more in your jupyter notebook.

Steps -

1. Install jupyter-themes -

   • using anaconda

     conda install -c conda-forge jupyterthemes

   • using pip

     conda install -c pip install jupyterthemes

2. Change the theme, cell width, cell height

   jt -t chesterish -cellw 100% lineh 170

• Data Science Hack #26 Parse_dates in read_csv() to change data type to datetime

What do you do when you need to change the data type of a column to DateTime? We can do this directly at the time of reading data using parse_dates argument.

• Data Science Hack 27 Share jupyter notebook using nbviewer

You can share your jupyter notebook with non-programmers very easily and the best way to do it is by using jupyter nbviewer. Pro tip - You can use Binder to execute the code from nbviewer on your machine!

• Data Science Hack #28 Plotting Decision Tree

Do you know how to plot a decision tree in just 1 line of code? Sklearn provides a simple function plot_tree() to do this task. You can tweak the hyperparameters as per your requirements.

• Data Science Hack #29 Invert Dictionary

Do you know how you can invert a dictionary in python? Dictionary is a collection which is unordered, changeable and indexed. It is widely used in day to day programming, and machine learning tasks.

• Data Science Hack #30 Interactive plots using plotly

Cufflinks binds plotly directly to pandas dataframes! Therefore you can make interactive charts without any hassle or long codes.

• Data Science Hack #31 Write python file directly from jupyter notebook cell

This hack is about saving contents of a cell to a .py file using the magic command %%writefile and then running the file in another jupyter notebook using the magic command %run

• Data Science Hack #32 Pretty-print Data structures

Are you getting confused while printing some of the data structures? Worry not, it is very common. pretty-print module provides an easy way to print the data structures in a visually pleasing way!

• Data Science Hack #33 Date Parser

This code allows you to convert date of any format into a specified format. Many times, we receive dates of various formats in our data. This hack will help you to convert all those formats into a specified format.

• Data Science Hack #34 Feature Selection using SelectFromModel

One of the ways to perform feature selection is by using feature_importance_ attribute of the base estimators. Using SelectFromModel function you can specify the estimator and the threshold for feature_importance_, This hack uses 'mean' as the threshold. You can tweak the threshold to get optimum results. To learn more visit the documentation

• Data Science Hack #35 Convert Strings into Characters

What could be the easiest way to convert a string to characters? Here is a simple hack which comes in handy while working with text data

• Data Science Hack #36 Resize Image Size

While building an image classification model using deep learning, it is required that all the images should be of same size. However, as the data comes from different sources, images may have different shapes. So, to convert them to same shape, we can use the resize function from open cv. This hack will will help you convert the images of any shape to a specified shape.

• Data Science Hack #37 Apply pandas in parallel

Does it take time to perform operations on your pandas dataframe? Pandarallel is a simple and efficient tool to parallelize Pandas operations on all your available CPUs!

• Data Science Hack #38 Generator Expressions vs List comprehension

The generator yields one item at a time and generates them only when in demand. Generators are much more memory efficient. This hack compares generator expressions with list comprehensions.

• Data Science Hack #39 Test your Regex

Do you avoid regex because they are hard to read and write as well as tricky to get right? This hack helps you get your regex correct. regex101 is an online regex tester, debugger with highlighting for PHP, PCRE, Python, Golang and JavaScript

• Data Science Hack #40 Convert List of Lists to List

Sometimes the data can be in the form of nested list. For example, the data can be date-wise transaction records for a particular product. However, you might need only in a single dimension. This hack will help you to flatten the list of lists into a single list.

• Data Science Hack #41 Hide Print Statements

We often use print statements for debugging purposes. This hack will help you to turn off print statements in a particular section of the code so that it will make debugging easier.

• Data Science Hack #42 Split PDF Document page-wise

This hack will help you to split a single PDF document into multiple pages.

• Data Science Hack #43 Merge PDF Documents

This hack will help you to combine multiple PDF documents into a single document. This hack is the inverse of Hack #42 Split PDF Document page-wise

• Data Science Hack #44 Create a Custom Image DataGenerator in Keras

Sometimes you would need a functionality which is not directly provided by Keras's ImageDataGenerator. You can easily create a wrapper around it to suit your needs.

1. For example, your usecase is that you have multi-input Deep Learning model like this

(i.e. a neural network which takes input from multiple data sources, and does a combined training on this data), and you want that the data generator should be able to handle the data preparation on the fly, you can create a wrapper around ImageDataGenerator class to give the required output.This notebook explains a simple solution to this usecase.

2. Another use case could be that you want to resize the images from a shape say 150x150 to a shape 224x224, which is generally utilized by the pretrained models, you can customize the ImageDataGenerator without coding your own data generator from ground up (Example Notebook).