Businesses can tap into the power of text analytics and natural language processing (NLP) to extract actionable insights from text data. Here’s how it works.

Text Analytics Basics

Text analytics (also known as text mining or text data mining) is the process of extracting information and uncovering actionable insights from unstructured text.

Text analytics allows data scientists and analysts to evaluate content to determine its relevancy to a specific topic. Researchers mine and analyze text by leveraging sophisticated software developed by computer scientists.

Example business use cases for text analytics include:

• Customer 360. Analyzing customer email, surveys, call center logs, and social media streams such as blogs, tweets, forum posts, and newsfeeds to understand customers better.
• Warranty analysis. Understanding text from dealer service professionals, warranty claims, orders, and similar sources.
• Product or service reviews. Analysis of customer reviews of products or services helps enterprises understand user sentiment or common issues customers are talking about.
• Recruitment. Keyword analysis (comparing profiles with job descriptions) helps in short-listing suitable candidates.

The Text Analytics Process

There are many ways text analytics can be implemented depending on the business needs, data types, and data sources. All share four key steps.

Step 1: Data Acquisition

Text analytics begins with collecting the text to be analyzed — defining, selecting, acquiring, and storing raw data. This data can include text documents, web pages (blogs, news, etc.), and online reviews, among other sources. Data sources can be internal or external to an organization.

Step 2: Data Preparation

Once data is acquired, the enterprise must prepare it for analysis. The data must be in the proper form to work with machine learning models that will be used for data analysis. There are four stages in data preparation:

• Text cleansing removes any unnecessary or unwanted information, such as ads from web pages. Text data is restructured to ensure data can be read the same way across the system and to improve data integrity (also known as “text normalization”).
• Tokenization breaks up a sequence of strings into pieces (such as words, keywords, phrases, symbols, and other elements) called tokens. Semantically meaningful pieces (such as words) will be used for analysis.
• Part-of-speech tagging (also referred as “PoS”) assigns a grammatical category to the identified tokens. Familiar grammatical categories include noun, verb, adjective, and adverb.
• Parsing creates syntactic structures from the text based on the tokens and PoS models. Parsing algorithms consider the text’s grammar for syntactic structuring. Sentences with the same meaning but different grammatical structures will result in different syntactic structures.

The Role of Natural Language Processing

NLP is a component of text analytics. Most advanced text analytics platforms and products use NLP algorithms for linguistic (language-driven) analysis that helps machines read text. NLP analyzes words for relevancy, including related words that should be considered equivalent, even if they are expressed differently (e.g., humor vs. humour). It’s the workhorse behind steps 2 and 3 described above.

One popular application of NLP is identifying relevant, quality content for search engines. For example, Google uses NLP in several ways, the most prominent of which is in search engine organization and categorization.

Long ago, a webmaster could achieve a higher rank in Google search results just by stuffing keywords into web content, so Google revised how its search engine processed content using numerous algorithms and NLP. NLP helps Google identify “spammy” content and categorize it. Google may de-index this content, penalize it, or simply rank it much lower than other content.

NLP is also used in email spam filters. Spammers try their best to evade such filters by changing words around, purposely misspelling words, or using synonyms. Email spam filters use a variety of factors to identify and block spam, phishing, and malicious content. Gmail’s filter, for example, incorporates machine learning and NLP to perform “sentiment analysis.” If content is determined to likely be spam, the content is sent to the user’s junk folder. For some content, Gmail deletes the message.

A decade ago, application of NLP was comparatively complicated. AI-based technologies (including NLP and text analytics) have evolved considerably, and there are many cloud services, commercial products, and open source platforms businesses can leverage. Here are few open source NLP applications:

• Stanford CoreNLP
• Natural Language Toolkit
• Apache Lucene and Solr
• Apache OpenNLP
• GATE and Apache UIMA

A Final Word

Text analytics isn’t new, but it is still unfamiliar to many organizations. With APIs, cloud-based AI services, and open source platforms available today, your business can leverage the power of text analytics to get a competitive edge by better understanding your customers and improving your brand’s value.

Published in TDWI UPSIDE.

What do you think?

One of the last subjects Stephen Hawking wrote was not as widely reported as perhaps it should have been. The physicist, who had previously warned about the potential threat artificial intelligence posed, more recently suggested that humans faced an even greater and more immediate threat. Sometime in the foreseeable future, he said, the human race could divide into two: those with an average intelligence level by today’s standards and those with super intelligence. The latter breed will have bodies improved by genetic engineering and brains improved by artificial intelligence (AI). These “superhumans”, as they are called, will be relatively few in number, but will pose a serious threat to normal humans.
If “Super Humans” exist in future, the super intelligent might be few in number because genetic engineering of humans’ brains and bodies will be very expensive, and only the very wealthy (or wealthiest) will be able to afford it. The result will be the gradual consigning of most humans to the role of a subservient class, less healthy and less intelligent than the others. In a few generations the superhumans’ progeny will begin to inherit the enhanced traits and, so, medical intervention to engineer those enhancements will become less necessary.
Some experts believe AI is a serious threat because machines will eventually become conscious, develop more sophisticated brains than humans, and decide humans are dispensable. Yet despite such dystopian warnings about AI, nobody knows if machines will ever become conscious, indeed nobody really knows what consciousness is. The idea of machines becoming smarter than humans and threatening the human race is pure speculation and most experts believe it’s unlikely to happen this century, if at all. The creation of superhuman beings, however, is less speculative. Already, humans can be improved by genetic engineering and most experts accept that greatly improving the human brain’s cognitive abilities by both genetic engineering and electronic implantation will happen sooner than most people think. Hawking suggests that anticipated advances in genetics will enable people to acquire improved memory and intelligence, as well as improved disease resistance and longer lifespans.
Talking about “Super Humans”, two solutions floating across internet are - To give every person the same chance to become superhuman. The second is to ban the technology. Neither of these solutions will work. First one is simply not practical, not affordable and could be the biggest threat for human race. Later one will just stop human evolution. History is not reassuring on either count.
In summary, AI/Robots might not be a treat to humanity nor will replace humans, at least in this century. AI will augment human workforce.
The use of AI and Robots should be highly regulated knowing the potential they have, to create negative impact to humans and world at large. Regulations should also help in restricting inventions/discoveries that are a treat to human race, like “Super Humans”.Article published on becominghuman.ai – https://becominghuman.ai/ai-robots-do-not-threaten-humans-but-super-humans-do-21c29ea455db

Read the full article on TDWI.

That’s a significant impact for businesses of all sizes. If you are an executive or a business owner who wants to put the power of AI-powered chatbots to work for your business, it is imperative you understand what is happening in the world of chatbots.

2019 looks to be the year when chatbots emerge as a viable business-building growth tool. Rather than creating a mobile app for their business, many businesses will opt for an easy-to-develop (and deploy) chatbot to connect with consumers.
The following are seven top chatbot trends you need to know before investing in building one.

1. Expect to see an explosion of the following
   1. BOTs, both chat and voice enabled.
   2. New use cases across all functions of an organization (some crazy ones too).
   3. New chatbots products, platforms and services.
   4. Market consolidation (M&As)
   5. Potentially new laws around how AI & chatbots should be used. Example: California B.O.T. Act of 2018, effective July 1st, 2018.
2. Chatbots and IoT are a match made in heaven. As more devices become internet-connected, expect to see an influx of IoT-enabled chatbots. Whether it is learning how to use your new wearable device or user-onboarding for enterprise clients managing an IoE (internet of everything) device, chatbots will play a critical role going forward.
3. There will be increasing usage of chatbots on company websites. While chatbots on social networking platforms like Facebook are already popular, a growing number of businesses want to “own” these conversations and keep communication happening on their website instead of a third-party platform.
4. Online businesses will discover the power and potential of chatbots for driving sales, improving customer satisfaction ratings, and customer education. Expect to see a growing number of developers offering chatbots for specific e-commerce platforms like Shopify. (apps.shopify.com/search/query?utf8=%E2%9C%93&q=chatbot).
5. Business leaders who want to increase the effectiveness of their sales funnel will start to use chatbots to a greater degree to appropriately funnel potential customers to the right departments. The more a company can learn about a consumer’s needs before speaking to a live person, the better they can help that consumer by connecting them with the right person from the start.
6. As artificial intelligence SDKs and APIs (programmableweb.com/search/artificial%20intelligence) become increasingly refined, expect to see an explosion of interest from companies who want to put the power of AI to work for their business. From IBM’s Watson to Apple’s Siri, the number of AI tools for developers is surging. The more tools savvy developers have at their disposal, the greater the products they can create for organizations on the hunt for chatbots.
7. Open-source code will play a significant role in the proliferation of chatbots. From Botpress.io to Rasa.com, there is already growth happening in the open-source chatbot sector. The more businesses develop a hunger for chatbots, the more significant impetus there will be for developers to code their creations.

Understanding upcoming chatbot trends is critical if you want to harness the power of chatbots for your company. The better you understand what the future holds for chatbots, the faster you can capitalize on a trend before your competitors.

Are Chatbots part of your 2019 growth strategy? If so, good luck in creating new conversational experience for your prospects and customers.
Article published on Chatbotslife.com – https://chatbotslife.com/chatbot-trends-you-need-to-know-before-investing-8cd037bea463

A simple definition of data science is that it’s the study of analyzing information and predicting outcomes. The predictions are mainly made using machine learning, but just one model can take months of data extraction, cleanup, coding, and deployment. Data science requires much larger reservoirs of data than a standard application using basic algorithms. You can’t use a few dozen stored records to analyze data accurately. You need millions of records to build and test a model. The first step for a data scientist to work with any organization is to gather and clean data. You’ve probably heard of “big data” and may even use the technology in your current applications. Big data is unstructured, but it’s perfect for data science. Unstructured data technologies grab as many records as possible and store them in a database such as MongoDB. This data can be anything, but just as an example consider a website and each of its pages. A crawler finds pages on a website and stores its text, images, and links in an unstructured record. You can scrape an entire site and get its data without worrying about structuring the data as you crawl as long as you use a database that supports unstructured formats. The next step is to clean the data, which is probably the most tedious part of data science. Most scientists clean the data and load it into a CSV file, which is a comma-delimited list of values. These files are easy to import either into another database or code, and any operating system supports After collecting the data, it’s time to figure out its functions. The data scientist first analyzes the data he has and asks a question. For instance, maybe you want to know what products are more likely to attract customers. You could take data from your e-commerce store and use previous customer orders to determine which products are most popular and which ones could be popular during the holidays to improve your sales and focus marketing efforts. Data science models could answer this question for you and make predictions using machine learning to contribute to improving your sales.

Building a Model

After the data scientist and the business determine the question to be answered, it’s time to build a model. A model is a unit of code that represents the “answer” to the question. The answer is usually represented in a graph to make it easier for the public to consume and understand the information. The visuals are typically a part of a library imported into the project, but the data scientist must ensure that the analysis that transforms data to a graph is acc. One of two main programming languages often is chosen to create the models. R is the language of math and statistics, so is the likely choice if your scientists have a mathematics educational background. More people understand Python, which is suitable for other development projects and is more popular among data scientists. Colleges teach Python, and because of its wide use within programming circles, you might find it easier to implement with a smaller learning. The data scientist creates the model with the question in mind. Using the e-commerce example, here’s how it works: The data scientist would review the data and set it up as rows and columns to import into Python code, which then calculates and displays it as a graph. The graph can be any number of plots, charts, and even visualization tools such as Excel or PowerPoint. The visual output is used to present information to the business for them to sign off on the results. Once the analysis is shown to be accurate, the data scientist can move on to the next step, which is creating the. The foundation for the model is logistic code that takes the data stored in a CSV file and runs it through the data scientist’s algorithms. The algorithms could be open-source or custom made by the scientist. It’s not uncommon for a developer to also dive into the analytics to better understand what must be dep. Although every model is different, you can just think of them as a module of code that represents the answer to a question. The business asks the question, and the data scientist develops a solution in the form of a model.

Model Deployment

One of the toughest parts of the entire data science workflow is deploying a model. The scientist must work with a developer to convert the model (written in R or Python) into the local application’s language, which can be any language such as C#, JavaScript, PHP, or VB. The developer might not be an analytics expert, and it’s up to the scientist to explain the model in a way that makes it easy for the developer to convert it. Statistically, most models never even make it to the application. Developers are hesitant to deploy machine learning code into an application without fully understanding what the code does, but it’s necessary for the business to take advantage of each. A good example of a machine learning module and local applications is in the finance industry. When you go to a cashier and want credit at the department store, you give her some information about yourself without telling your life story. From just your social security number, name and birth date, an algorithm can decide whether or not to give you credit. These financial decisions are made using data science models. The question that the model answers is: “Are you a trustworthy candidate for credit, and does your data history indicate that you will pay the money back?” You know that the response to a credit application is either yes or no. Data models written by data scientists make this decision, and they use machine learning to pick up on patterns within millions of records.

Integrating Data Science into Business Code

Building a new data team is costly, takes time, and there is a learning curve for your developers. The benefits far outweigh the disadvantages, and you can work with project managers and agencies to help get you started.

If you’ve thought of taking your business analytics to the next level, adding a data scientist to your current IT team is the way to go. Your developers will learn new skills, your business will make more money, and you can take advantage of the latest in code design and database storage technologies.

Google Dialogflow (formerly api.ai) is one of the best platforms to build bots (a.k.a agents). Dialogflow provides interesting out-of-the-box features like inline editor for fulfillment, one-click integrations, small talk, and more. Chatbots built on Dialogflow can be delivered via multiple communication platforms like Facebook messenger, Telegram, Twilio, Skype, Google Assistant and Kik using one-click integrations provided by Dialogflow. Explore all integrations here: https://cloud.google.com/dialogflow-enterprise/docs/integrations/

Use cases that demand integrating Dialogflow chatbots into websites or web applications would require a chat app that seamlessly integrates with Dialogflow. That is Dataflix Messenger (DM), a chat app designed for humans to interact with chatbots built on Google Dialogflow. Businesses can integrate Dialogflow agents (chatbots) into websites and web applications using DM, delivering rich conversational experiences. Key Features

• Guided conversation with quick replies
• Seamless integrations – websites, apps, portals
• Fully hosted, server less architecture
• Deploy anywhere – cloud or on-prem
• Two step configuration with Google Dialogflow
• Easy customization and re-banding

WordPress Integration Dataflix Messenger app can be integrated into WordPress websites using simple Java code that comes with DM, and plugins like “Insert Header and Footer”. Check out the live demo at the bottom of this page (right). Configuring Dataflix Messenger to your Dialogflow agent and integrating chatbot into your website is a simple three step process –

1. Host DM in the cloud or in your data center. DM can be hosted in any cloud service that allows hosting static websites, including Google Cloud Storage and AWS S3. SSL needs to be enabled to ensure communication between the browser and website is secure and encrypted.
2. Configure DM to integrate with a Dialogflow agent using Client Access Token and Developer Access Token. Both the tokens are specific to an agent, and can be found under “Settings” tab in the Dialogflow console.
3. Chatbot can be integrated into any website or web application using Java code / iframe that comes with DM and URL from step 2.

Interested in exploring and deploying Dataflix Messenger in your website for no cost?

• Adding comments to dashboards.
• Users managing tables (pricing, etc.).
• Audit applications.
• Feedback forms.

How can this be accomplished with Looker?
By using Actions by Looker.

What should you be familiar with?                                

• Python (or NodeJS)
• LookML

What should you have?

• Permissions to manage Google Cloud Functions in your GCP.
• Looker Dev Access

Have them all? Let’s get started.

Example use case:

Consider the following table:

actions_ad_data

Users should be able to update the information/add comments in this table directly from Looker dashboards. This article walks you through the process of achieving that functionality.

As a part of that, to catch all the entered comments, a new table has to be created on the database, similar to this one:

actions_comments

And on Looker, create a view on the comments table, and join it with the initial ad_data explore, as follows:

view: actions_comments {
  sql_table_name: !@#$%^&*.actions_comments ;;
  dimension: ad_id {
    type: number
    sql: ${TABLE}.ad_id ;;
  }
  dimension: comment {
    type: string
    sql: ${TABLE}.comment ;;
  }
  dimension: p_key {
    type: string
    primary_key: yes
    sql: ${ad_id}|| '' || ${time_raw} ;;
  }
  dimension_group: time {
    type: time
    timeframes: [
      raw,
      time,
      date,
      week,
      month,
      quarter,
      year
    ]
    sql: ${TABLE}.time ;;
  }
  dimension: u_name {
    type: string
    sql: ${TABLE}.u_name ;;
  }
  measure: count {
    type: count
    drill_fields: [u_name]
  }
}

connection: "!@#$%^&*"
include: "*.view.lkml"
explore: actions_ad_data {
  label: "Ad Data"
  description: "Created just to try Data Actions."
join: actions_comments {
  sql_on: ${actions_ad_data_dt.ad_id} = ${actions_comments.ad_id} ;;
  relationship: one_to_many
  type: left_outer
}
}

Before moving on with actual implementation, here is a high-level illustration of the entire writeback process using Data Actions:
Within a Looker Action, all the parameters and forms are bundled together as a JSON. And when a cloud function is triggered by that action, that cloud function collects the bundled information from Looker, validates it and posts the data onto the database according to the given instructions. Make sure to give those instructions good and accurate.

Now, assuming that all of the important visualizations will be including the actions_ad_data.ad_id, two data actions were added to that dimension, as follows:

view: actions_ad_data {
  sql_table_name: !@#$%^&*.actions_ad_data ;;
.
.
.
dimension: ad_id {
    type: number
    primary_key: yes
    sql: ${TABLE}.ad_id ;;

   action: {
      label: "Update Money Spent"
      url: "https://!@#$%^&*.cloudfunctions.net/changeamount_actions"
     param: {
        name:"id"
        value: "{{value}}"
      }
     form_param: {
        name: "changeamount"
        type: string
        label: "Amount"
        description: "Enter new amount above."
        required: yes
      }
    }


   action: {
      label: "Add comment"
      url: "https://!@#$%^&*.cloudfunctions.net/changeamount_comments"
     param: {
        name:"id"
        value: "{{value}}"
      }
     user_attribute_param: {
        user_attribute: user_attribute_name
        name: "username"
      }
     form_param: {
        name: "comment"
        type: textarea
        label: "Enter the comment"
        required: yes
      }
    }
  }
.
.
.
}

The two actions are “Update Money Spent” and “Add comment”. They are almost the same but understand that the target URLs are different, targeting a cloud function relevant to the action. Here are the cloud functions that are up and running on the Google Cloud:
Here is “changeamount_actions” function configuration:

Note: Don’t forget to mention ‘google-cloud-bigquery==1.5.0’ in requirements.txt of each cloud functions. Also, this function can be done using NodeJS as well.

And it’s source code:

from google.cloud import bigquery     # DO NOT FORGET THIS
import datetime
import time

def changeamount(request):
r = request.get_json() # Fetch the data action JSON

client = bigquery.Client()
dataset_id = '!@#$%^&*' # Replace with name of your BQ dataset
table_id = 'actions_ad_data'  # replace with your table ID
table_ref = client.dataset(dataset_id).table(table_id)
table = client.get_table(table_ref)

# getting the data
id = r['data']['id']
amount = r['form_params']['changeamount']
sys_time = int(time.time())

row_to_update = [
(
id,
amount,
datetime.datetime.fromtimestamp(sys_time).strftime('%Y-%m-%d %H:%M:%S')
)
]

row = client.insert_rows(table, row_to_update)  # API request to insert row
return '{"looker": {"success": true,"refresh_query": true}}' # return success response to Looker forcing it to refresh

[/vc_column_text][vc_column_text css=”.vc_custom_1541804171616{margin-bottom: 0px !important;}”]The other Cloud Function’s configuration is similar to the first one, and so here is the source code of it:

from google.cloud import bigquery
import datetime
import time

def addcomment(request):
r = request.get_json() # Fetch the data action JSON

client = bigquery.Client()
dataset_id = 'learning' # Replace with name of the BQ dataset 
table_id = 'actions_comments' # replace with your table ID
table_ref = client.dataset(dataset_id).table(table_id)
table = client.get_table(table_ref) # API request

# request variables
id = r['data']['id']
u_name = r['data']['username']
comment = r['form_params']['comment']
sys_time = int(time.time())

row_to_update = [
(
comment,
u_name,
datetime.datetime.fromtimestamp(sys_time).strftime('%Y-%m-%d %H:%M:%S'),
id
)
]
row = client.insert_rows(table, row_to_update) # API request to insert row
return '{"looker": {"success": true,"refresh_query": true}}' # return success response to Looker

Breaking down the code:

Take a look at the “Update Money Spent” action declaration in actions_ad_data.

Within that action, param and form_param contain the data that might be required to be sent to the server (which in this case is the Cloud Function). Read about param and form_param in this link.

The second action is “Add comment”, which is similar to the first one, but with a new user_attribute_param, which just contains the user attributes, and in this case the full name of currently logged in user. This can also be achieved by using ‘liquid’.

Now, on the cloud function, the dataset_id and table_id onto which the data is to be written is hard-coded into specific variables. And also, the function tries to retrieve data from the JSON, store it within appropriate variables and then tries inserting all the variables together onto the table as a new row.

Here is a video of Data Actions on an example dashboard created on the example data:

Remember:

#1

BigQuery is append-only. So, new rows will be added, instead of updating existing rows. Here is one way to avoid showing duplicates on Looker:

• Make sure the table have timestamps.
• Create a PDT to retrieve the latest information of each pkey, and set a sql_trigger on the timestamp field.

#2

Do not share the Cloud Function’s URL. Anyone with the URL can write data to your BigQuery instance. Try adding some checks/validation that the request is coming from Looker.

#3

Make sure the end-user knows where the data actions are, and how they’re supposed to be used.

Other Useful links:

Joins in Looker: https://docs.looker.com/data-modeling/learning-lookml/working-with-joins

About Datagroups: https://docs.looker.com/reference/model-params/datagroup

Another example: https://discourse.looker.com/t/update-write-back-data-on-bigquery-from-looks/9408/2

This blog was developed in collaboration with Looker’s Customer Success Engineer Jesse Carah.