I’ve been working away on a coding challenge to match records from two different data sources, and I think I’m close to being done. I haven’t done much work involving processing text documents, so it’s been a learning experience. Given more time I’d like to look into classifying listings based on their sentence structure, but the weather is getting nice enough that I’m going to have put side projects on the back burner for a bit ☺.

GitHub Repo

I built a Naive Bayes classifier listings as being either for a camera or an accessory. To generate the training set, I’m using a heuristic classifier I developed earlier and then going through the results manually.

Steps to run:
1) Create a preliminary training set using a heuristic based classifier. Only a randomly selected subset of the listings should be used to build the training set. As there are far more camera listings than accessory listings, it’s important to reflect that ratio in the training set.
2) Manually go through the training data to remove any entries. Try to remove incorrect classifications in pairs to keep the ratio of camera listings to accessory listings the same.
3) Run the Bayes classifier using the training set you created. To not be considered a camera a listing must contain more than 2 “accessory” words and more than 90% of the words need to be “camera” words.

The +/- signs prepended to each term show how they contributed to a listing being classified as a camera

Positives:

Negatives:

False positives:

Source Code

This is my second attempt at generating entity aliases from a set of product listings. In my first attempt I tried to use MinHash to group similar product listings, but it didn’t work out. This time, I’m going to be joining on n-grams of the manufacturer name and model number and then scoring the joins by the inverse term probability of the model n-grams.

I’m planning to use this in a pre-processing stage so that when I block by canonical manufacturer names I can include listings that have different variations of the canonical name.

For each listing, I’m attempting to find a similar canonical product record. The criteria I’m using for something being similar are:

1. Similar manufacture using character n-gram similarity
2. Similar model using shingles made from model parts

If I find enough instances where a listing is similar to a canonical product, I’ll consider the listing’s manufacturer name to be an alias for the canonical product’s manufacturer name.

Here are some aliases I was able to find:

My main goal was to relate listings with a manufacturer of fuji to fujifilm so it looks like this is going to work. I had to change the scoring method to use the inverse term probably because some of the model names have common numbers or words (ex: “zoom”) which caused lots of false positives.

Messy Source Code

A follow along workshop on using R to do data mining on tweets.

Notes

• Data pipeline: Extract → Clean → Transform → Analyize
• Clean
  • Remove: Unicode emoji, punctuation, links, stop words, numbers, dates, URLs, white space
  • Normalize to lowercase
  • Stemming - Truncate words to their radicals
    • Examples: cats → cat, ponies → poni
• Analysis
  • Part of speech tagging - Tag each word with its part of speech in a sentence using definition and context.
    • Example: They refuse to permit -> [pronoun] [verb] [to] [verb]
  • Word association - Dot product of two words in a term document matrix gives correlation coefficient.
  • Clustering - Grouping similar tweets(docs) together
    • K-means - Centroidal model, each cluster is represented by a single mean vector.
      • Algorithm: create random clusters → assign points to nearest cluster centroid → recalculate cluster centroids to the average of assigned data points → repeat.
    • Hierarchical - Connectivity model
    • Use cosine similarity as the distance function. Cosine similarity normalizes for tweet(doc) length during the comparison.
  • Topic Mining - Use probability of terms to discover information from documents. Documents may belong to multiple topics.

Word association example:

Correlation(word 2, word 3) = DotProduct([1,0,0,0], [1,1,0,0]) = 0.5

Meetup Event