This article is the second part of a series. You can find the first article here.
Do you need Machine Learning?
The truth is, not every problem requires machine learning.
Some problems are better solved with traditional rule-based programming approaches. So, how do you know whether or not you need machine learning? Here is a check-list I use that you might find useful:
Do you want to automate the task?
Can you formulate the problem clearly?
Do you have sufficient examples?
Does the problem have a regular pattern?
Can you find meaningful representations of the data?
How do you define success?
Do you want to automate the task?
It just so happens that not everything should be automated. Some things need a human touch. Good candidates for automation are tasks that include high volumes of data, such as if you need to deal with customer emails for an online store that is doing really well. Secondly, the data should be complex and unstructured, again like in an email. Thirdly, the rules for processing the data should be complex at best, and possibly even hard to define. If your problem meets those criteria, then it is a good candidate for Machine Learning.
In the example above, what you have could be a sentiment analysis problem, where you try to find out whether your customers are leaving positive or negative reviews based on the emails they send in.
It could also be a classification problem. Suppose you run a really busy airline and customers send emails with complaints such as lost baggage, damaged baggage, flight reschedules, or some other complaint. Because of the large number of customers you may have to deal with you can’t have people reading each individual email to find out what the problem is before escalating to the right department. Instead, you could automatically sort the mails as they come in so that customer support specialists deal with only the right type of support request.
Can you formulate the problem clearly?
Looking at our sentiment analysis example, the question becomes: what is the input, and what is the output? As we can see from that example, the input is a review text from a customer, and the output is a label of positive, negative, or neutral.
Do you have sufficient data?
Machine Learning is all about data. The more data you have, the better. Also, for a supervised learning problem, your data must be labeled, meaning that some specialists would have labeled the data correctly before it is used for training.
Does the problem have a regular pattern?
Machine learning learns patterns from your data. While it would be difficult or even impossible for a human to read through millions of records to find a pattern, that task is trivial for machines.
For our sentiment analysis example, the machine would find words that are frequently associated with positive reviews, and words that are frequently associated with negative reviews.
Can you find meaningful representations of the data?
While this is not meant to scare you, the truth is that machines represent everything as numbers. In order to process data, you must find a way of meaningfully representing that data as numbers.
For sentiment analysis, the data are represented as feature vectors, which is a set of numbers. A good feature representation underlies the success of any Machine Learning process.
How do you define success?
This is very important. A metric must exist to measure the performance of the Machine Learning system. These metrics are called evaluation criteria. If this metric meets or exceeds your expectations, then the system can be put into use. If not, then the system needs to be discarded. For example, if your sentiment classifier has an accuracy of 95% and your business only needs 80% accuracy, then it is successful.
With the above checklist, you should be closer to evaluating whether or not a problem is suitable for Machine Learning.
Please leave a comment with your thoughts. Thanks for reading, and stay tuned for the next article in this series. Also, if you subscribe, you can get notifications of new articles.
This article was originally published on my Medium blog.