Sentiment Analysis on News Articles using Python

Sentiment Analysis on News Articles using python

by Milind Paradkar

In our previous post onsentiment analysiswe briefly explained sentiment analysis within the context of trading, and also provided a model code in R. The R model was applied on an earnings call conference transcript of an NSE listed company, and the output of the model was compared with the quarterly earnings numbers, and by charting the one-month stock price movement post the earnings call date. QuantInsti also conducted a webinar on “Quantitative Trading Using Sentiment Analysis” where Rajib Ranjan Borah , Director & Co-founder, iRageCapital and QuantInsti, covered important aspects of the topic in detail, and is a must watch for all enthusiast wanting to learn & apply quantitative trading strategies using sentiment analysis.

Taking these initiatives on sentiment analysis forward, in this blog post we attempt to build a Python model to perform sentiment analysis on news articles that are published on a financial markets portal. We will build a basic model to extract the polarity (positive or negative) of the news articles.

In Rajib’s Webinar, one of the slides details the sensitivity of different sectors to company and sectorial news. In the slide, the Pharma sector ranks at the top as the most sensitive sector, and in this blog we will apply our sentiment analysis model on specific news articles pertaining to select Indian Pharma companies. We will determine the polarity, and then check how the market reacted to these news. For our sample model, we have taken ten Indian Pharma companies that make the NIFTY Pharma index.

Building the Model

Now, let us dive straight in and build our model. We use the following Python libraries to build the model:

Requests Beautiful Soup Pattern Step 1: Create a list of the news section URL of the component companies

We identify the component companies of the NIFTY Pharma index, and create a dictionary in python which contains the company names as the keys, while the dictionary values comprise the respective company abbreviation used by the financial portal site to form the news section URL. Using this dictionary we create a python list of the news section URLs for the all components companies.

importcsv importtime importrequests frombs4importBeautifulSoup frompattern.enimportngrams Base_url = "http://www.moneycontrol.com" # Build a dictionary of companies and their abbreviated names companies = {'cadilahealthcare':'CHC','piramalenterprises':'PH05', 'glenmarkpharma':'GP08','glaxosmithklinepharmaceuticals':'GSK', 'sunpharmaceuticalindustries':'SPI','lupinlaboratories':'LL', 'cipla':'C','aurobindopharma':'AP', 'drreddyslaboratories':'DRL','divislaboratories':'DL03'} # Create a list of the news section urls of the respective companies url_list = ['http://www.moneycontrol.com/company-article/{}/news/{}#{}'.format(k,v,v) for k,v in companies.iteritems()] printurl_list Step 2: Extract the relevant news articles web-links from the company’s news section page

Using the Python list of the news section URLs, we run a Python For loop which pings the portal with every URL in our Python list. We use the requests.get function from the Python requests library(which is a simple HTTP library). The requests module allows you to send HTTP/1.1 requests. One can add headers, form data, multipart files, and parameters with simple Python dictionaries, and also access the response data in the same way.

The text of the response object is then applied to create a Beautiful Soup object. Beautiful Soup is a Python library for pulling data out of HTML and XML files. It works with a given parser to provide for ways of navigating, searching, and modifying the parse tree.

HTML parsing basically means taking in the HTML code and extracting relevant information like the title of the page, paragraphs in the page, headings, links, bold text etc.

The news section webpage on the financial portal site contains 20 news articles per page. We target only the first page of the news section, and our objective is to extract the links for all the news articles that appear on the first page using the parsed HTML. We inspect the HTML, and use the find_all method in the code to search for a tag that has the CSS class name as “arial11_summ”. This enables us to extract all the 20 web-links.

Fortunes of the R&D intensive Indian Pharma sector are driven by sales in the US market and by approvals/rejections of new drugs by US Food and Drug Administration (USFDA). Hence, we will select only those news articles pertaining to the US Food and Drug Administration (USFDA) and the US market. Using keywords like “US”, “USA”, and “USFDA” in a If statement which is nested within the Python For Loop, we get us our final list of all the relevant news articles.

# Create an empty list which will contain the selected news articles List_of_links = [] # Extract the relevant news articles weblinks from the news section of selected companies for urlsin url_list: html = requests.get(urls) soup = BeautifulSoup(html.text,'html.parser') # Create a BeautifulSoup object # Retrieve a list of all the links and the titles for the respective links word1,word2,word3 = "US","USA","USFDA" sub_links = soup.find_all('a', class_='arial11_summ') for linksin sub_links: sp = BeautifulSoup(str(links),'html.parser')# first convert into a string tag = sp.a if word1in tag['title'] or word2in tag['title'] or word3in tag['title']: category_links = Base_url + tag["href"] List_of_links.append(category_links) time.sleep(3) # Print the select list of news articles weblinks #for p in List_of_links: print p Step 3: Remove the duplicate news articles based on news title

It may happen that the financial portal publishes important news articles pertaining to the overall pharma sector on every pharma company’s news section webpage. Hence, it becomes necessary to weed out the duplicate news articles that appear in our Python list before we run our sentiment analysis model. We call the set function on our Python list which we generated in Step 2 to give us a list with no duplicate news articles.

# Remove the duplicate news articles based on News Title unique_links = list(set(List_of_links)) for q in unique_links: print q # Create a dictionary of positive/negative words related to the Pharma Sector reader = csv.reader(open('dict.csv', 'r')) pharma_dict = dict((rows[0],rows[1]) for rowsin reader) # Creating an empty list which will be filled later with news article links, and Polarity values (pos/neg) df =[] printdf Step 4: Extract the main text from the selected news articles In this step we run a Python For Loop and for every news article URL, we call the requests.get() on the URL, and then convert the text of response object into a Beautiful Soup object. Finally, we extract the main text using the find and get_text methods f