Sentiment Analysis By Retrieving Data From Web Page

In this we have retrieving data form web pages and perform the following task:

TextBlob Module

wordcloud Module

*conda install -c conda-forge wordcloud

Newspaper3k

TextTastic Module

https://www.rev.com/blog/transcripts/donald-trump-white-house-rally-speech-transcript-october-10-first-event-since-covid-diagnosis

Importing All Related Libraries

import requests        # import from web
 from bs4 import BeautifulSoup      # clean up text
 from wordcloud import WordCloud    # create word clouds
 from textblob import TextBlob      # basic NLP, install first
 from textatistic import Textatistic   # readability, install first
 
 from pathlib import Path    # for quick import of text file for NLP
 
 import pandas as pd
 import seaborn as sns
 import matplotlib.pyplot as plt
 from plotly import express as px
 
 # Magics
 %config InlineBackend.figure_format = 'retina'
 %matplotlib inline

Example 1: Extraction Block or Not Allowed

## 403 Forbidden Error, extract blocked / not allowed
 url = 'https://www.americanrhetoric.com/speeches/mlkihaveadream.htm'
 
 response = requests.get(url)   # retrieve the webpage
 response.content               # show content from the retrieved page

Example 2: Static, Predominately Text-based Web-Page

## Static web page - JFK speech re: moon
 url = 'https://er.jsc.nasa.gov/seh/ricetalk.htm'
 
 response = requests.get(url)
 response.content       # Notice moderate amount of HTML code

soup = BeautifulSoup(response.content, 'html5lib')
 text = soup.get_text(strip=True)  # text without tags
 text       #BeautifulSoup has done a decent job on this page removing HTML

Example 3: Sometimes it is easier to copy and paste to a file

## Somewhat hidden text
 url = 'https://www.whitehouse.gov/briefings-statements/remarks-president-trump-2020-united-states-military-academy-west-point-graduation-ceremony/'
 
 response = requests.get(url)
 response.content            # UGLY! significant amount of code -- Where's the text??

## Soup doesn't help much in this case
 soup = BeautifulSoup(response.content, 'html5lib')
 text = soup.get_text(strip=True)  
 text

Steps: Install newspaper3k via pip (only do this once per machine)

Now you are ready for other tasks (view text, check authors and publication date; perform NLP tasks).

https://newspaper.readthedocs.io/en/latest/

Try the newspaper article code on your own link to a site that is likely to have all of the atributes

# Change the url to your own, comment out all urls but one, note may not show all of article if behind pay wall
 url = 'https://hbr.org/2020/04/bringing-an-analytics-mindset-to-the-pandemic'
 # url = 'https://www.wsj.com/articles/ceos-increasingly-see-sustainability-as-path-to-profitability-11602535250'
 # url = 'https://www.cnn.com/2020/10/13/health/us-coronavirus-tuesday/index.html'

article = Article(url)
 article.download()
 article.parse()

print("Title: ", article.title)  
 print("Authors: ", article.authors)  
 print("Publication Date: ", article.publish_date)  
 print("First Image:", article.top_image)  
 print("Video Links:", article.movies)

print("Title: ", article.title)
 print()
 print(article.text)

article.nlp()
 
 print("KeyWords: ", article.keywords)   # creates a list of authors; no authors on this page
 print()
 print("Summary: ", article.summary)  # no publish date on this web-page

# Set the url
 
 # From rev.com
 url = 'https://www.rev.com/blog/transcripts/donald-trump-mosinee-wi-rally-speech-transcript-september-17'
 event = '-mosinee-2020' # this will be part of the file name for a text file we create 
 
 #url = 'https://www.rev.com/blog/transcripts/ruth-bader-ginsburg-stanford-rathbun-lecture-transcript-2017'
 #event = '-standfordlecture-2017'

# Minimum code needed to get to the text of the speech
 article = Article(url)
 article.download()
 article.parse()
 print(article.text) 

# write the text to a file
 
 with open('speech.txt', 'w') as f:
     f.writelines(text)
 
 with open('speech.txt', 'r') as f:
  for cnt, line in enumerate(f):
  print(f'Line {cnt}: {line}')

# Custom processing for rev site
 # line 0 = speaker and time
 # line 2 = what speaker said
 # lines 1 and 3 = blanks
 # create four lists of the components of speech
 
 with open('speech.txt', 'r') as f:
     speech = f.readlines()
 
 tmp = []
 speaker = []
 time = []
 words = []
 
 for cnt, line in enumerate(speech):
  if cnt % 2 == 0:
         tmp.append(line.rstrip())   # temp list of just the text lines 0,2
 
 for i in range(0,len(tmp),2):
     speaker.append(tmp[i].split(': ')[0])  #split speaker line into 2 parts
     time.append(tmp[i].split(': ')[1])
     words.append(tmp[i+1])    # words from speaker

# find unique speaker names for later filter
 
 set(speaker)

spkr = 'Donald Trump'
 file = spkr.split()[len(spkr.split())-1] + event + '.txt'
 file

# use write instead of writelines since we don't want entire list
 # remember to add new line
 
 with open(file,'w') as f:
  for i in range(0,len(speaker)):
  if speaker[i] == spkr:
             f.write(words[i]+'\n')

# Confirm good file
 text = Path(file).read_text()
 text

Web scrapping using beautiful soup is used in machine learning or data science to extract useful data and preform machine learning algorithms or task on it, like: sentiment analysis in NLP etc.

If need any help related to this then contact us at: contact@codersarts.com