As a part of any data cleanup process you’re probably going to want to normalize your data.
Like in golf, if you’re playing against Tiger Woods you’ll want to give him a handicap, just so that he won’t totally blow you away and that the game might still be fun.
Also with data you may want all things equal.
Assume you’re trying to compare the height of a building and it’s internal temperature. The two different units are fine, except when you're trying to teach a machine about the data you may want the units to have some sort of equal measure. (300 meters vs. 26 degrees. To us that makes sense, but the actual value of the digits is clearly incongruous.)
There are many different ways of doing this. Personally I like the MinMaxScalar function in the sklearn package. First off it gives you pretty straight-forward results (everything ends up being between 0 and 1), and secondly, it's the only one I know how to use so far.
So take it or leave it my pretties...
Let’s make a numpy array. Here are 6 tuples of fictional height/temperature data:
buildingData = np.array([[300,24],[200,21],[126,18],[567,27],[420,19],[189,30]]) print (buildingData)
[[300 24] [200 21] [126 18] [567 27] [420 19] [189 30]]
As I said we'll be using sklearn to do this stuff, so first you’ll need to import the MinMaxScalar function:
from sklearn.preprocessing import MinMaxScaler
Then we need to figure out the largest and smallest data point in your data set:
scaler_model = MinMaxScaler() scaler_model.fit(buildingData)
Then scale the data appropriately:
scaled_data = scaler_model.transform(buildingData)
This will take the highest and lowest values in your data, turn them into 1 and 0 respectively, then stuff all the other values into relative numbers between 1 and zero. So we get:
print (scaled_data) = [[ 0.39455782 0.5 ] [ 0.16780045 0.25 ] [ 0. 0. ] [ 1. 0.75 ] [ 0.66666667 0.08333333] [ 0.14285714 1. ]]
Voila, all of your data is (proportionately speaking) the same as it was before, but reducing it into relative values between 0 and 1 will allow a machine to process it appropriately.