Web Data Processing Systems Project

Part 1: Python Wrapper For Spark

The basic operation of the PySpark_Parser, is to

extract text data from compressed WARC files
Pre-process and sanitize the data for further analysis
Extract any active entities
Perform Entity Linking/Disambiguation with a Knowledge Base(Freebase)
Resort only on the baseline ranking system of the Freebase(No time for self-ranking methods)
Parallelize the main procedure function by performing SPark map-reduce operations

Part 2: Perform Entity Extraction and Analysis based on New York Times Articles

The basic operation of the Entity Analysis - New York Times.py, is to

Perform data extraction from New York Times API
Recognise any of the unique entities and their frequency per document
Recognise any relational patterns between entities(Co-occurances)
Perform Sentiment Analysis on Entity content
Perform basic statistics and measure any Influence points among the entities
Depict the results in a form of bar chart