Awesome Open Source
Awesome Open Source

Table of contents

  1. Project description
  2. Results and learnings
    2.1. Main challenge and initial assumptions
    2.2. Dataset
    2.3. Training and evaluation results
    2.4. Model deployment and usage
  3. Run the example
    3.1. Prerequisites
    3.2. Train and evaluate the model
    3.3. Deploy web service
  4. Code highlights

1. Project description

[back to the top]

This case study shows how to create a model for text analysis and classification and deploy it as a web service in Azure cloud in order to automatically classify support tickets.
This project is a proof of concept made by Microsoft (Commercial Software Engineering team) in collaboration with Endava.
Our combined team tried 3 different approaches to tackle this challenge using:

What will you find inside:

  • How to clean and prepare text data and featurize it to make it valuable for machine learning scenarios
  • How to strip the data from any sensitive information and also anonymize/encrypt it
  • How to create a classification model using Python modules like: sklearn, nltk, matplotlib, pandas
  • How to create a web service with a trained model and deploy it to Azure
  • How to leverage Azure Machine Learning Service to make working on ML projects easier, faster and much more efficient

The team:


2. Results and learnings

[back to the top]

Disclaimer: This POC and all the learnings you can find below is an outcome of close cooperation between Microsoft and Endava. Our combined team spent total of 3 days in order to solve a challenge of automatic support tickets classification.

2.1. Main challenge and initial assumptions

[back to the top]

  • Main challenge we tried to solve was to create a model for automatic support tickets classification for Endavas helpdesk solution. As Endava stated: currently helpdesk operators waste a lot of time evaluating tickets and trying to assign values to properties like: ticket_type, urgency, impact, category, etc. for each submitted ticket
  • The dataset we used is Endavas internal data imported from their helpdesk system. We were able to collect around 50k classified support tickets with original messages from users and already assigned labels
  • In our POC we focused only on tickets submited in form of an email, similar to the one below:

2.2. Dataset

[back to the top]

  • For the sake of this repository, data have been stripped out of any sensitive information and anonymized (encrypted). In the original solution we worked on a full dataset without any encryptions. You can download anonymized dataset from here.

  • Example of anonymized and preprocessed data from AML Workbench view:

    [!Important] Azure Machine Learning service no longer supports the deprecated Workbench tool.

  • Below you can see a sample data transformation flow we used while preparing our dataset:

  • After evaluating the data we quickly discovered that distribution of values for most of columns we wanted to classify is strongly unbalanced with some of the unique values represented by even as low as 1-2 samples. There are multiple techniques to deal with that kind of issues but due to limited amount of time for this POC we were not able to test them in action.

  • Distribution of values for each column:

    ticket_type business_service
    impact urgency
    category sub_category1
    sub_category2

2.3. Training and evaluation results

[back to the top]

In order to train our models, we used Azure Machine Learning Services to run training jobs with different parameters and then compare the results and pick up the one with the best values.:

To train models we tested 2 different algorithms: SVM and Naive Bayes. In both cases results were pretty similar but for some of the models, Naive Bayes performed much better (especially after applying hyperparameters) so at some point we decided to work with NB only.

Below you can find some of the results of models we trained to predict different properties:

  • ticket_type

    We started from predicting the least unbalanced (and most important from Endavas business point of view) parameter which is ticket_type and after training the model and finding the best hyperparameters using GridSearchCV (which improved precision and recall by around 4%), we were able to achieve some really good results which you can see below:

    confusion matrix for ticket_type metrics for ticket_type
  • business_service

    business_service property is one of the unbalanced features with very low amount of samples/per class for most values.
    We started from running the training on a subset of our dataset where we removed business_service values which were represented by less then 100 samples.
    Unfortunately that didn't help much and we still had a lot of classes that were not at all recognized. So we continued to increase the minimum required number of samples per class untill we started to see some meaningful results:

    confusion matrix for business_service metrics for business_service
  • category, impact and urgency

    To predict category, impact and urgency we took the same approach as with business_service property but results looked even worse. It's obvious that such level of unbalance within the data makes it impossible to create a model with any meaningful results.
    If you would only look at mean/average value of precision and recall you could wrongly assume that results are quite well but if you would check the values of support for each class it would become clear that because one class which covers 70-90% of our data, the results are completely rigged:

    confusion matrix for category metrics for category
    confusion matrix for impact metrics for impact
    confusion matrix for urgency metrics for urgency

2.4. Model deployment and usage

[back to the top]

Final model will be used in form of a web service running on Azure and that's why we prepared a sample RESTful web service written in Python and using Flask module. This web service makes use of our trained model and provides API which accepts email body (text) and returns predicted properties.

You can find a running web service hosted on Azure Web Apps here: https://endavaclassifiertest1.azurewebsites.net/.
The project we based our service on with code and all the deployment scripts can be found here: karolzak/CNTK-Python-Web-Service-on-Azure.

Sample request and response in Postman: Demo


3. Run the example

3.1. Prerequisites

[back to the top]

  • Download content of this repo

    You can either clone this repo or just download it and unzip to some folder

  • Setup Python environment

    In order to run scripts from this repo you should have a proper Python environment setup. If you don't want to setup it locally you can use one of the Data Science Virtual Machine images (both on Linux and Windows) on Azure. All of them come in with most popular data science and machine learning tools and frameworks already preinstalled and ready for you.

  • Install dependencies

    Make sure to install all the dependencies for this project. You can easily do it by using requirements.txt file and running this command:

    pip install -r requirements.txt
    

    Please report issues if you find any errors or missing modules, thanks!

  • Download Endava support tickets dataset (all_tickets.csv)

    You can download the dataset from here or by executing 1_download_dataset.py script. If you decide to download it manually, just make sure to put it under:

    project
    └───datasets
        └───all_tickets.csv
    

    Endavas support tickets dataset is already cleaned and stripped out of any unnecessary words and characters. You can check some of the preprocessing operations that were used in 0_preprocess_data.py script.

3.2. Train and evaluate the model

[back to the top]

To train the model you need to run 2_train_and_eval_model.py script. There are some parameters you could possibly play around with - check out code highlights section for more info.

3.3. Deploy web service

[back to the top]

Inside webservice folder you can find scripts to setup a Python based RESTful web service (made with Flask module).

Deeper in that folder you can also find download_models.py script which can be used to download some already trained models that will be used by the web service app.

In order to deploy it to an environment like Azure App Service you can check this GitHub repo for some inspiration.


4. Code highlights

[back to the top]

  • 0_preprocess_data.py - collection of scripts we used for data preprocessing and anonymization. I attached it in case someone would be interested in steps we followed with our data

    In order to clean our data we removed:

    • headers and footers
    • email metadata (like: from, to, cc, date, etc.)
    • email addresses, phone numbers and urls
    • image references
    • blacklisted words (Endavas sensitive information)
    • non-english words - few percent of emails contained Romanian language
    • all numerical values
    • all non-alphabetic characters
    • whitespaces

    In order to anonymize the dataset for publishing purposes we used sklearn.preprocessing.LabelEncoder:

    def encryptSingleColumn(data):
        le = preprocessing.LabelEncoder()
        le.fit(data)
        return le.transform(data)
    
    
    def encryptColumnsCollection(data, columnsToEncrypt):
        for column in columnsToEncrypt:
            data[column] = encryptSingleColumn(data[column])
        return data
    
  • 1_download_dataset.py - simple script used to download our dataset (already preprocessed, cleaned and anonymized)

  • 2_train_and_eval_model.py -

    • column_to_predict variable is used to determine which column should be used for classification

      column_to_predict = "ticket_type"
      # Supported datasets:
      # ticket_type
      # business_service
      # category
      # impact
      # urgency
      # sub_category1
      # sub_category2
      
    • You can play around with some variables in order to improve accuracy:

      classifier = "NB"  # Supported algorithms: "SVM" and "NB"
      use_grid_search = False  # grid search is used to find hyperparameters. Searching for hyperparameters is time consuming
      remove_stop_words = True  # removes stop words from processed text
      stop_words_lang = 'english'  # used with 'remove_stop_words' and defines language of stop words collection
      use_stemming = False  # word stemming using nltk
      fit_prior = True  # if use_stemming == True then it should be set to False ?? double check
      min_data_per_class = 1  # used to determine number of samples required for each class. Classes with less than that will be excluded from the dataset. Default value is 1
      
    • Loading dataset into pandas.DataFrame object:

          
      # dfTickets = package.run('AllTickets.dprep', dataflow_idx=0) 
      
      # loading dataset from csv
      dfTickets = pd.read_csv(
          './datasets/all_tickets.csv',
          dtype=str
      )  
      
    • Splitting dataset using sklearn.model_selection.train_test_split:

      # Split dataset into training and testing data
      train_data, test_data, train_labels, test_labels = train_test_split(
          data, labelData, test_size=0.2
      )  # split data to train/test sets with 80:20 ratio
      
    • You can use one of 3 different count vectorizers for features extraction from text:

      # Extracting features from text
      # Count vectorizer
      if remove_stop_words:
          count_vect = CountVectorizer(stop_words=stop_words_lang)
      elif use_stemming:
          count_vect = StemmedCountVectorizer(stop_words=stop_words_lang)
      else:
          count_vect = CountVectorizer()
      
    • Creating a Pipeline to organize transforms and final estimator and fitting the dataset into it:

      text_clf = Pipeline([
          ('vect', count_vect),
          ('tfidf', TfidfTransformer()),
          ('clf', MultinomialNB(fit_prior=fit_prior))
      ])
      text_clf = text_clf.fit(train_data, train_labels)
      
    • sklearn.model_selection.GridSearchCV can be used to search for best possible set of parameters for the learning algorithm:

      if use_grid_search:
          # Grid Search
          # Here, we are creating a list of parameters for which we would like to do performance tuning.
          # All the parameters name start with the classifier name (remember the arbitrary name we gave).
          # E.g. vect__ngram_range; here we are telling to use unigram and bigrams and choose the one which is optimal.
          
          # NB parameters
          parameters = {
              'vect__ngram_range': [(1, 1), (1, 2)],
              'tfidf__use_idf': (True, False),
              'clf__alpha': (1e-2, 1e-3)
          }
      
          # SVM parameters
          #    'vect__max_df': (0.5, 0.75, 1.0),
          #    'vect__max_features': (None, 5000, 10000, 50000),
          #    'vect__ngram_range': ((1, 1), (1, 2)),  # unigrams or bigrams
          #    'tfidf__use_idf': (True, False),
          #    'tfidf__norm': ('l1', 'l2'),
          #    'clf__alpha': (0.00001, 0.000001),
          #    'clf__penalty': ('l2', 'elasticnet'),
          #    'clf__n_iter': (10, 50, 80),
          # }
      
          # Next, we create an instance of the grid search by passing the classifier, parameters
          # and n_jobs=-1 which tells to use multiple cores from user machine.
          gs_clf = GridSearchCV(text_clf, parameters, n_jobs=-1)
          gs_clf = gs_clf.fit(train_data, train_labels)
      
          # To see the best mean score and the params, run the following code
          gs_clf.best_score_
          gs_clf.best_params_
      
    • Predicting labels for test set, evaluating accuracy of the model (with and without GridSearch) and printing out a simple confusion matrix:

      print("Evaluating model")
      # Score and evaluate model on test data using model without hyperparameter tuning
      predicted = text_clf.predict(test_data)
      prediction_acc = np.mean(predicted == test_labels)
      print("Confusion matrix without GridSearch:")
      print(metrics.confusion_matrix(test_labels, predicted))
      print("Mean without GridSearch: " + str(prediction_acc))
      
      # Score and evaluate model on test data using model WITH hyperparameter tuning
      if use_grid_search:
          predicted = gs_clf.predict(test_data)
          prediction_acc = np.mean(predicted == test_labels)
          print("Confusion matrix with GridSearch:")
          print(metrics.confusion_matrix(test_labels, predicted))
          print("Mean with GridSearch: " + str(prediction_acc))
      
    • Plotting confusion matrix using heatmap from seaborn module:

      # Ploting confusion matrix with 'seaborn' module
      # Use below line only with Jupyter Notebook
      # %matplotlib inline
      import seaborn as sns
      from sklearn.metrics import confusion_matrix
      import matplotlib.pyplot as plt
      import matplotlib
      mat = confusion_matrix(test_labels, predicted)
      plt.figure(figsize=(4, 4))
      sns.set()
      sns.heatmap(mat.T, square=True, annot=True, fmt='d', cbar=False,
                  xticklabels=np.unique(test_labels),
                  yticklabels=np.unique(test_labels))
      plt.xlabel('true label')
      plt.ylabel('predicted label')
      # Save confusion matrix
      # plt.savefig(os.path.join('.', 'outputs', 'confusion_matrix.png'))
      plt.show()
      

      Resulting confusion matrix should look similar to this:

    • Printing out classification report:

      # Printing classification report
      # Use below line only with Jupyter Notebook
      from sklearn.metrics import classification_report
      print(classification_report(test_labels, predicted,
                                  target_names=np.unique(test_labels)))
      

      Resulting classification report should look similar to this:

    • Serializing trained models using pickle module:

      # Save trained models to /output folder
      if use_grid_search:
          pickle.dump(
              gs_clf,
              open(os.path.join(
                  '.', 'outputs', column_to_predict+".model"),
                  'wb'
              )
          )
      else:
          pickle.dump(
              text_clf,
              open(os.path.join(
                  '.', 'outputs', column_to_predict+".model"),
                  'wb'
              )
          )
      
  • webservice.py -

    • Loading pretrained models from serialized files using pickle:

      model_ticket_type = pickle.load(
          open(
              os.path.join(__location__, "ticket_type.model"), "rb"
          )
      )
      
    • Extracting description text from requests json, fitting it into the model to get a prediction and returning the result as json:

      @app.route('/endava/api/v1.0/tickettype', methods=['POST'])
      def tickettype():
          ts = time.gmtime()
          logging.info("Request received - %s" % time.strftime("%Y-%m-%d %H:%M:%S", ts))
          print(request)
          print(request.json)
          if not request.json or 'description' not in request.json:
              abort(400)
          description = request.json['description']
          print(description)
      
          predicted = model_ticket_type.predict([description])
          print("Predicted: " + str(predicted))
      
          ts = time.gmtime()
          logging.info("Request sent to evaluation - %s" % time.strftime("%Y-%m-%d %H:%M:%S", ts))
          return jsonify({"ticket_type": predicted[0]})
      
Alternatives To Support Tickets Classification
Select To Compare


Alternative Project Comparisons
Related Awesome Lists
Top Programming Languages
Top Projects

Get A Weekly Email With Trending Projects For These Topics
No Spam. Unsubscribe easily at any time.
Python (796,107
Learning (76,469
Machine Learning (36,527
Artificial Intelligence (18,756
Azure (17,753
Classification (13,254
Microsoft (11,031
Pandas (6,539
Numpy (5,859
Webservices (3,330
Text Classification (1,542
Text Mining (917
Text Analysis (857
Text Processing (612
Azure App Services (159
Azure Machine Learning (74
Azure Webapp (52
Azure Web App Service (11