Skip to main content

Data Wrangling — Some Tips during this covid season

In this covid season, thought of sharing some tips and sample codes on Data Wrangling. After getting a glance of Google Colab, my go to language for data wrangling has been Python. This blog hence uses Colab and Pandas on Python for data wrangling.

Most of the data collected has format related issues because they are collected without the use of a software. These days, with tools like Google Forms coming into picture, it is much better, but when you have old data to deal with in its row form, there is no escape from formatting it and getting it to the shape that you need it to be in.

This blog shows how to use colab, pandas and lambda functions to quickly format some data.

I had written an earlier blog few days back on  and how to import data from excel to Colab. So, I shall skip that part now.

Let us say, the below is the data that we need to format (the admission number column) and we need all the data to be in the format: YYYY-Num

Image for post

We can do this quickly using dataframes and lambda functions in Python.

The below is what we can do to do the transformation:

Import the data into a dataframe using pandas.

Then, write a normal function in Python which will replace the slash in a string with a “-”. Later, apply a lambda function on the column “Admission Number” and pass this function which we just wrote.

We can repeat the same with another function to format the date. In the second function we can split the data with “-” and check for the format of the year and replace it if required in the YYYY format. Apply this lambda function again on the column which requires the data to be formatted.

The resulting data can be exported to another .csv on the drive too. A piece of demo code is written to show this. The note book has been uploaded to my Github from Colab directly and is available . Below are the screenshots of the relevant pieces of code.

Image for post
Image for post

Might sound simple. But, these simple tools worked like magic on my old Mac with performance issues to transform 1000s of records and to get some “visualisable” data quickly.


Originally posted here

Comments

Post a Comment

Popular posts from this blog

Top Tips For Presenting Data Visualizations

By [https://EzineArticles.com/expert/Rob_Ian_Chapman/575203]Rob Ian Chapman So you have played around with your data, made some key discoveries that are going to revolutionize your company and now need to persuade others to join your enlightened brigade. The key is turning your analysis into an effective and persuasive presentation. Here are some tips for showing your data visualizations to the world: Planing Decide whether you are going to start with a conclusion and then explain (deductive approach) or build the story through to the conclusion (inductive approach). Think about your visualizations only once you know you want to present them Write out the point of each visualization slide before designing it For each conclusion from your analysis, try several different visualizations before deciding on the best one to display your data If your conclusion is particularly contentious draw it from multiple visualizations building the story as you go, rather than attempti...
Post a Comment
Read more

How Data Visualization Helps Enhance the Value of Business Intelligence

How Data Visualization Helps Enhance the Value of Business Intelligence By Frank Poladi With massive amounts of data available both internally and externally, making sense of the information isn't easy. However, business intelligence (BI) tools make it easier. With a robust BI solution in place, it becomes possible to mine data from diverse databases for insights, trends, and analysis. Business intelligence is hot right now, but not all BI solutions are created equally. As powerful as these tools are and as richly detailed as reports may be, the results can be difficult for the average manager to decipher - and more importantly, use. Fortunately, it's not necessary to puzzle over stacks of business intelligence reports when you have a solution that includes data visualization tools. What is data visualization? Visualizations are a graphical way of displaying data such as pie charts, bar charts, and trend lines. In addition to these familiar charts and graphs, data vi...
Post a Comment
Read more

The Science of Today's Technology, Data Science

The Science of Today's Technology, Data Science The Science of Today's Technology, Data Science By Shalini M Technology today... Recently, there has been a surge in the consumption and innovation of information based technology all over the world. Every person, from a child to an 80-year-old man, use the facilities the technology has provided us. Along with this, the increase in population has also played a big role in the tremendous growth of information technology. Now, since there are hundreds of millions of people using this technology, the amount of data must be large too. The normal database software like Oracle and SQL aren't enough to process this enormous amount of data. Hence the terms 'Big data' and 'Data science' were coined. Big data has made quite an impact on the world and data science has recently risen to be one of the hottest topics. Now how are these two related? What is data science? It is the field of science where different...
Post a Comment
Read more