1. Searching for gold inside HTML files¶
It used to take days for financial news to spread via radio, newspapers, and word of mouth. Now, in the age of the internet, it takes seconds. Did you know news articles are automatically being generated from figures and earnings call streams? Hedge funds and independent traders are using data science to process this wealth of information in the quest for profit.
In this notebook, we will generate investing insight by applying sentiment analysis on financial news headlines from FINVIZ.com. Using this natural language processing technique, we can understand the emotion behind the headlines and predict whether the market feels good or bad about a stock. It would then be possible to make educated guesses on how certain stocks will perform and trade accordingly. (And hopefully, make money!)
Why headlines? And why from FINVIZ?
- Headlines, which have similar length, are easier to parse and group than full articles, which vary in length.
- FINVIZ has a list of trusted websites, and headlines from these sites tend to be more consistent in their jargon than those from independent bloggers. Consistent textual patterns will improve the sentiment analysis.
As web scraping requires data science ethics (sending a lot of traffic to a FINVIZ's servers isn't very nice), the HTML files for Facebook and Tesla at various points in time have been downloaded. Let's import these files into memory.
Disclaimer: Investing in the stock market involves risk and can lead to monetary loss. The content in this notebook is not to be taken as financial advice.
# Import libraries
from bs4 import BeautifulSoup
import pandas as pd
import os
html_tables = {}
# For every table in the datasets folder...
for table_path in os.listdir('datasets'):
# Read the contents of the file into 'html'
html = BeautifulSoup(open('datasets/' + table_path, 'r'))
# Find 'news-table' in the Soup and load it into 'html_table'
html_table = html.find(id='news-table')
# Add the table to our dictionary
html_tables[table_path] = html_table
2. What is inside those files anyway?¶
We've grabbed the table that contains the headlines from each stock's HTML file, but before we start parsing those tables further, we need to understand how the data in that table is structured. We have a few options for this:
- Open the HTML file with a text editor (preferably one with syntax highlighting, like Sublime Text) and explore it there
- Use your browser's webdev toolkit to explore the HTML
- Explore the headlines table here in this notebook!
Let's do the third option.
# Get all the rows in Tesla's headlines table
tsla = html_tables['tsla_22sep.html']
tsla_tr = tsla.findAll('tr')
# For each row in Tesla's headlines table
for i, table_row in enumerate(tsla_tr):
# Read the text of the element 'a' into 'link_text'
link_text = table_row.a.get_text()
# Read the text of the element 'td' into 'data_text'
data_text = table_row.td.get_text()
# Print the count
print(f'{i}:')
# Print the text
print(link_text)
print(data_text)
# Exit the loop after three rows
if i == 3:
break
3. Extra, extra! Extract the news headlines¶
As we saw above, the interesting data inside each table row (<tr>) is in the text inside the <td> and <a> tags. Let's now actually parse the data for all tables in a comfortable data structure.
# Hold the parsed news into a list
parsed_news = []
# Iterate through the news
for file_name, news_table in html_tables.items():
# Iterate through all tr tags in 'news_table'
for x in news_table.findAll('tr'):
# Read the text from the tr tag into text
text = x.get_text()
headline = x.a.get_text()
# Split the text in the td tag into a list
date_scrape = x.td.text.split()
# If the length of 'date_scrape' is 1, load 'time' as the only element
# If not, load 'date' as the 1st element and 'time' as the second
if len(date_scrape) == 1:
time = date_scrape[0]
else:
date = date_scrape[0]
time = date_scrape[1]
# Extract the ticker from the file name, get the string up to the 1st '_'
ticker = file_name[0:file_name.find('_')]
# Append ticker, date, time and headline as a list to the 'parsed_news' list
parsed_news.append([ticker, date, time, headline])
print(parsed_news[0])
print(parsed_news[1])
4. Make NLTK think like a financial journalist¶
Sentiment analysis is very sensitive to context. As an example, saying "This is so addictive!" often means something positive if the context is a video game you are enjoying with your friends, but it very often means something negative when we are talking about opioids. Remember that the reason we chose headlines is so we can try to extract sentiment from financial journalists, who like most professionals, have their own lingo. Let's now make NLTK think like a financial journalist by adding some new words and sentiment values to our lexicon.
# NLTK VADER for sentiment analysis
from nltk.sentiment.vader import SentimentIntensityAnalyzer
# download VADER lexicon
# nltk.download('stopwords')
# New words and values
new_words = {
'crushes': 10,
'beats': 5,
'misses': -5,
'trouble': -10,
'falls': -100,
}
# Instantiate the sentiment intensity analyzer with the existing lexicon
vader = SentimentIntensityAnalyzer()
vader.lexicon.update(new_words)
5. BREAKING NEWS: NLTK Crushes Sentiment Estimates¶
Now that we have the data and the algorithm loaded, we will get to the core of the matter: programmatically predicting sentiment out of news headlines! Luckily for us, VADER is very high level so, in this case, we will not adjust the model further* other than the lexicon additions from before.
*VADER "out-of-the-box" with some extra lexicon would likely translate into heavy losses with real money. A real sentiment analysis tool with chances of being profitable will require a very extensive and dedicated to finance news lexicon. Furthermore, it might also not be enough using a pre-packaged model like VADER.
# Use these column names
columns = ['ticker', 'date', 'time', 'headline']
# Convert the list of lists into a DataFrame
scored_news = pd.DataFrame(parsed_news, columns=columns)
# Iterate through the headlines and get the polarity scores
scores = scored_news.headline.apply(vader.polarity_scores)
# Convert the list of dicts into a DataFrame
scores_df = pd.DataFrame.from_records(scores)
# Join the DataFrames
scored_news = scored_news.join(scores_df)
# Convert the date column from string to datetime
scored_news['date'] = pd.to_datetime(scored_news.date).dt.date
6. Plot all the sentiment in subplots¶
Now that we have the scores, let's start plotting the results. We will start by plotting the time series for the stocks we have.
import matplotlib.pyplot as plt
plt.style.use("fivethirtyeight")
%matplotlib inline
# Group by date and ticker columns from scored_news and calculate the mean
mean_c = scored_news.groupby(['date','ticker'])['compound'].mean()
# Unstack the column ticker
mean_c = mean_c.unstack('ticker')
# Get the cross-section of compound in the 'columns' axis
mean_c = mean_c.xs(['fb','tsla'], axis='columns')
# Plot a bar chart with pandas
mean_c.plot.bar()
7. Weekends and duplicates¶
What happened to Tesla on November 22nd? Since we happen to have the headlines inside our DataFrame, a quick peek reveals that there are a few problems with that particular day:
- There are only 5 headlines for that day.
- Two headlines are verbatim the same as another but from another news outlet.
Let's clean up the dataset a bit, but not too much! While some headlines are the same news piece from different sources, the fact that they are written differently could provide different perspectives on the same story. Plus, when one piece of news is more important, it tends to get more headlines from multiple sources. What we want to get rid of is verbatim copied headlines, as these are very likely coming from the same journalist and are just being "forwarded" around, so to speak.
# Count the number of headlines in scored_news (store as integer)
num_news_before = scored_news.headline.count()
# Drop duplicates based on ticker and headline
scored_news_clean = scored_news.drop_duplicates(subset=['ticker', 'headline'])
# Count number of headlines after dropping duplicates
num_news_after = scored_news_clean.headline.count()
# Compare before and after
print('before: ' + str(num_news_before) + '\n' + 'after: ' + str(num_news_after))
8. Sentiment on one single trading day and stock¶
Just to understand the possibilities of this dataset and get a better feel of the data, let's focus on one trading day and one single stock. We will make an informative plot where we will see the smallest grain possible: headline and subscores.
# Set the index to ticker and date
single_day = scored_news_clean.set_index(['ticker', 'date'])
# Cross-section the fb row
single_day = single_day.xs('fb')
# Select the 3rd of January of 2019
single_day = single_day.xs('2019-01-03')
# Convert the datetime string to just the time
single_day['time'] = pd.to_datetime(single_day['time']).dt.time
# Set the index to time and sort by it
single_day = single_day.set_index('time').sort_index()
9. Visualize the single day¶
We will make a plot to visualize the positive, negative and neutral scores for a single day of trading and a single stock. This is just one of the many ways to visualize this dataset.
TITLE = "Positive, negative and neutral sentiment for FB on 2019-01-03"
COLORS = ["red", "orange", "green"]
# Drop the columns that aren't useful for the plot
plot_day = single_day.drop(['compound', 'headline'], axis=1)
# Change the column names to 'negative', 'positive', and 'neutral'
plot_day.columns = ['negative', 'positive', 'neutral']
# Plot a stacked bar chart
plot_day.plot.bar(stacked = True, figsize = (10, 6), title = TITLE, color = COLORS)
