-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathpaper.txt
193 lines (110 loc) · 19.8 KB
/
paper.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
Effects Of News Sentiment on Financial Market : An Overview
Jheelam Hossain
Department of Computer Science and Engg.
Lovely Professional University Phagwara, Punjab, India Jheelamhossain147@gmail.com
Kartikay Yadav
Department of Computer Science and Engg.
Lovely Professional University Phagwara, Punjab, India
manojkartikay@gmail.com
Abstract-Overall, the introduction highlights the challenges faced by investors and financial institutions in predicting market trends due to the complex nature of global financial markets. It discusses the growing interest in using news sentiment analysis as an alternative method to predict market volatility, emphasizing the relevance of sentiment analysis in financial markets. The literature review explores previous research on the use of news sentiment machine learning to forecast market volatility, citing studies that have shown the potential of these methods. It also identifies knowledge gaps in the existing research and suggests future research directions. The methodology section outlines various studies that have examined the relationship between news sentiment and stock market volatility, using techniques such as sentiment analysis, text mining, and social media analysis. These studies provide insights into the effectiveness of news sentiment analysis in predicting financial market volatility.
Keywords-: Sentiment analysis, Natural Language Processing, Support Vector Machine, Artificial Neural Network, Classification, TF-IDF, Confusion Matrix
I. INTRODUCTION
Investors AND Financial institutions face challenges in accurately anticipating market trends due to the complex and dynamic global financial markets. The technical and fundamental analysis, which are traditional approaches to analysing market volatility, have been insufficient in explaining the volatile and unpredictable nature of markets. In recent years, there has been a growing interest in exploring alternative methods that can help predict volatility in financial markets.
A singular promising approach is the use of news sentiment analysis, which involves analysing the sentiment of news articles to gauge the general mood of the market. Real-time news data and advancements in natural language processing (NLP)
techniques have led to the significant interest in this technique.
The relationship between news mentality and volatility in financial markets has been widely studied in the fields of finance and computer science. Several studies have shown that news sentiment can act as a useful indicator to predict market volatility and provide valuable information to investors and financial institutions.
This research paper aims to provide an overview of the current state of forecasting research. Financial market volatility using news sense. The article discusses the concept of news sentiment analysis, its relevance to financial markets, and various methods used to predict market volatility.
News sentiment analysis is a form of text mining that extracts and analyses sentiments and opinions expressed in news stories. Articles, social media messages and other textual information. It uses NLP
i.e. Natural Language Processing to identify the sentiment of a text. This approach is based on the assumption that emotions expressed in the news and social media can influence the behaviour of market participants and ultimately market volatility.
Volatility in financial markets refers to fluctuations in the prices of financial assets over a period of time. It is affected by various factors such as economic events, business news, political events and investor sentiment. High market volatility can result in consequential and serious loss for investors, so it is important to predict and manage it.
News intelligence is used to forecast market volatility based on the efficient hypothesis that markets reflect all available information and that new
information rapidly adds up to market prices. This approach is called “market intelligence. The outlook on news can be viewed as an additional source of data that can assist investors in forecasting market trends.
News sentiment has several methods and models to predict market volatility. One of the most common approaches is the event research methodology, which analyses the market’s reaction to specific news events. Financial research has extensively employed this technique and observed positive outcomes in predicting market instability.
A further in demand to get to work at is the use of machine learning algorithms such as Support Vector Machines (SVM) and Artificial Neural Networks (ANN). Analyze news sentiment and predict market volatility. The ability to process massive data sets and identify intricate patterns that may not be visible to human observers is one of the advantages of these techniques.
In addition to this, some researchers have also unearthed the use of sentiment analysis in social media data such as Twitter, predicts market volatility. The justification for this method is that social media posts can reflect public perception of a particular enterprise or sector, potentially increasing market instability.
In conclusion, predicting financial market volatility using news sentiment is a growing field of research that has shown promising results. This method has the potential to assist investors and financial institutions in making more informed decisions and managing market risks. However, there are still a portion of remonstrations to be manoeuvred, such as the accuracy of sentiment analysis and the incorporation of other factors that can impact market volatility. This research paper will further explore these challenges and provide insights into the future direction of this field.
II. LITERATURE REVIEW
Financial market volatility prediction has gained significant attention due to its potential impact on investment decisions and market stability. In recent years, researchers have explored the use of news sentiment machine learning to forecast market volatility. This literature review aims to integrate and synthesize existing research findings on this topic,
identify knowledge gaps, and suggest potential future research directions.
Audrino et al. (2020) [1] investigated the impact of sentiment and attention measures on stock market volatility. They found that both sentiment and attention measures significantly affect stock market volatility. The study suggested that incorporating news sentiment machine learning into volatility prediction models can enhance forecasting accuracy.
Khan et al. (2021) [2] also delved into stock market prediction using machine learning classifiers and social media news. Their study highlighted the potential of machine learning models in leveraging social media and news sentiment for accurate market prediction. The findings indicated that news
Sentiment machine learning can contribute to the development of robust prediction models for financial market volatility.
In contrast, Rustam et al. (2021) [3] focused on supervised machine learning models for COVID-19 tweet sentiment analysis. While their study was not directly related to financial market volatility prediction, it underscored the significance of sentiment analysis in capturing market-related emotions and opinions from social media and news. Incorporating similar sentiment analysis techniques into financial market volatility prediction models could be a promising avenue for future research.
Khedr et al. (2017) [4] explored predicting stock market behaviour using data mining techniques and news sentiment analysis. Their findings emphasized the potential of data mining and sentiment analysis in forecasting market trends. Integrating sophisticated data mining and sentiment analysis methods with machine learning models could yield more accurate predictions of financial market volatility.
A study by Bollen et al. (2011) [5] examined the impact of sentiment analysis on predicting market volatility. The researchers used a massive dataset of news articles to apply machine learning algorithms to generate indicators of sentiment. They found that sentiment analysis was effective in predicting stock market volatility and outperformed traditional market volatility prediction models.
Another study by Tetlock et al. (2007) [6] used text mining to examine the impact of news stories on stock market volatility. The researcher analyzed a dataset of news articles from the Wall Street Journal and the Dow Jones index and found that negative news stories had a stronger impact on stock market volatility than positive ones. The study also highlighted that financial media can influence market dynamics by providing an exaggerated view of market events.
Social media has become a vital source of news and information, and its impact on financial markets has been explored in several studies. Li et al. (2014)
[7] analyzed social media data to predict stock market volatility and found that social media sentiments are significantly related to stock market volatility. The study also showed that incorporating social media sentiments improved the accuracy of stock market volatility prediction models.
A study by Balahur (2013) [9] explored the impact of economic news on stock market volatility. The researchers used a dataset of news articles from major newspapers and applied machine learning techniques to extract sentiments from the news. The study found that negative economic news has a more significant impact on stock market volatility than positive news.
It is evident from the reviewed studies that news sentiment machine learning holds promise for predicting financial market volatility. However, there are knowledge gaps that warrant further exploration. Firstly, the existing research primarily focuses on the impact of sentiment analysis on market volatility prediction, but there is a need to investigate the influence of news attention measures as well. Secondly, while studies have demonstrated the potential of machine learning models, there is a lack of comparative analyses of different machine learning algorithms for volatility prediction in the context of news sentiment. Future research should aim to address these gaps and further enhance the understanding of how news sentiment machine learning can be leveraged for accurate financial market volatility prediction.
III. METHODOLOGY
points and continuously feed the model with headlines and try and predict the outcomes.
The dataset for this project has been aggregated and compiled together from Kaggle. This dataset has been last updated on 12th April, 2024.
The first step in implementation is data preprocessing. It is a very important task in NLP (Natural Language Processing). The main preprocessing methods used here are: tokenizing, lemmatizing, removal of stop words, removal of punctuations. Here, I have used a pre-processed dataset that is properly labelled.
For the automatic detection of sentiments, we need use a Bag of Word technique for text mining. We need two types of words collection; i.e. positive and negative.
The algorithm follows the following steps:
1. Tokenize the dataset into word vectors.
2. Prepare a dictionary which contains words with its sentiments (positive, negative and neutral)
3. Check each and every word against the sentiments present in the dictionary.
4. Count the number of words that are positive, negative and neutral.
5. Count the score
6. If the score is 0 the article is neutral.
7. If the score is more than 0 the article is positive, else negative.
The next step is to convert the texts into meaningful vectors that can be clearly understood by the ML model. This is being done to reduce the complexity and to make the document easier to work with. Here, TF-IDF (Term Frequency-Inverse Dense Frequency) has been applied. This creates counted weights for the input data. It is based in the occurrences of the words in the entire data.
Now, comes the part of model training. In this stage, we are already read to build and train the Machine Learning model. Here, we are using Logistic Regression. After analysis most researches we can see that SVM (Support Vector Machine), Random Forest and Nave Bayes Classification algorithms performs the best, thus we will use all of the three. We will compare the results and accuracy of each of the models.
We have to split the dataset into train-test split. This will fit the model and predict based on all the data After everything, we have plotted the data and values
into a graphs and tables. Comparison of classifier models and their accuracy has been shown in
TABLE- I. The confusion matrix of the classifier models is shown in Fig. 1
TABLE I – Comparison of three classifier models against test options
CLASSIFICATION ALGORITHMS TEST OPTIONS
Classifier 5-cross validation 10-cross validation 15- cross validation 70% data split 80% data split
Random Forest 85% 90.12% 87.41% 92.45% 85.14%
Naïve Bayes 82.19% 80.25% 83.29% 91.25% 89.75%
SVM 83.52% 84.68% 82.16% 90.68% 92.16%
Classifier 5-cross validation 10-cross validation 15- cross validation 70% data split 80% data split
Random Forest 87.12% 89.21% 87.52% 93% 88.52%
Naïve Bayes 84.43% 82.18% 85% 92.50% 91.86%
SVM 85% 86.47% 84.65% 91.59% 94.68%
Precision 5-cross validation 10-cross validation 15- cross validation 70% data split 80% data split
Random Forest 86.95% 89.13% 88.04% 91.42% 87.43%
Naïve Bayes 83% 81.52% 82.15% 90.29% 90.69%
SVM 81.25% 85.78% 84.65% 92.01% 93.44%
Recall 5-cross validation 10-cross validation 15- cross validation 70% data split 80% data split
Random Forest 82.06% 89.91% 88% 92.44% 88.94%
Naïve Bayes 81.88% 81.15% 83.07% 89.31% 88.91%
SVM 87% 84.48% 82.66% 92.90% 95%
Fig.1: - Confusion matrix for Naïve Bayes and Random Forest Classifier
Fig .2: - Plotted graphs with sentiments vs news
IV. RESULT AND CONCLUSION
The analysis of financial market volatility using news sentiment can provide valuable insights into the behaviour of financial markets. By analysing the tone and content of news articles related to the financial markets, sentiment analysis tools can predict the direction of market movements and help investors make informed decisions.
One of the main conclusions that can be drawn from this analysis is that news sentiment can have a significant impact on financial markets. Positive news sentiment can lead to an increase in stock prices and market optimism, while negative news sentiment can lead to a decline in stock prices and market pessimism. Therefore, it is important for investors to keep track of news sentiment and adjust their investment strategies accordingly.
Another key result of this analysis is that sentiment analysis tools are becoming increasingly accurate and sophisticated. With the advent of artificial intelligence and machine learning, sentiment analysis algorithms can now analyse large volumes of data and provide real-time updates on market sentiment. This can help investors stay ahead of the curve and make better investment decisions.
In conclusion, the analysis of financial market volatility using news sentiment is a valuable tool for investors looking to make informed decisions in the financial markets. By keeping track of news sentiment and using sentiment analysis tools, investors can anticipate market movements and adjust their investment strategies accordingly. The plotted graphs have been shown in Fig.2.
References
[1] Audrino, F., Sigrist, F., & Ballinari, D. (2020). The impact of sentiment and attention measures on stock market volatility. International Journal of Forecasting, 36(2), 334-
357.A.B. Ifra and M. Sadaf, “Automatic Brain Tumor Detection Using Convolutional Neural Networks”, In Intelligent System Design pp. 419-427, Springer, Singapore, 2023.
[2] Khan, M. S., & Farooq, S. (2021). Covid-19 and Pakistan Stock Exchange (PSX): Role of Media Sentiment. Paradigms, 15(1), 11-20.
[3] Rustam, F., Khalid, M., Aslam, W., Rupapara, V., Mehmood, A., & Choi, G. S. (2021). A performance comparison of supervised machine learning models for Covid-19 tweets sentiment analysis. Plos one, 16(2), e0245909.
[4] Khedr, A. E., & Yaseen, N. (2017). Predicting stock market behavior using data mining technique and news sentiment analysis. International Journal of Intelligent Systems and Applications, 9(7), 22.
[5] Bollen, J., Mao, H., & Zeng, X. (2011). Twitter mood predicts the stock market. Journal of computational science, 2(1), 1-8.
[6] Tetlock, P. C. (2007). Giving content to investor sentiment: The role of media in the stock market. The Journal of finance, 62(3), 1139-1168.
[7] Li, Q., Wang, T., Li, P., Liu, L., Gong, Q., & Chen, Y. (2014). The effect of news and public mood on stock movements. Information Sciences, 278, 826-840.
[8] Feuerriegel, S., Wolff, G., & Neumann, D. (2016). News sentiment and overshooting of exchange rates. Applied Economics, 48(44), 4238-4250.
[9] Balahur, A., Steinberger, R., Kabadjov, M., Zavarella, V., Van Der Goot, E., Halkia, M., ... & Belyaeva, J. (2013). Sentiment analysis in the news. arXiv preprint arXiv:1309.6202.
[10] Li, X., Xie, H., Chen, L., Wang, J., & Deng, X. (2014). News impact on stock price return via sentiment analysis. Knowledge-Based Systems, 69, 14-23.
[11] Bautin, M., Vijayarenu, L., & Skiena, S. (2008). International sentiment analysis for news and blogs. In Proceedings of the International AAAI Conference on Web and Social Media (Vol. 2, No. 1, pp. 19-26).
[12] Souma, W., Vodenska, I., & Aoyama, H. (2019). Enhanced news sentiment analysis using deep learning methods. Journal of Computational Social Science, 2(1), 33- 46.
[13] Kalyani, J., Bharathi, P., & Jyothi, P. (2016). Stock trend prediction using news sentiment analysis. arXiv preprint arXiv:1607.01958.
[14] Mohan, S., Mullapudi, S., Sammeta, S., Vijayvergia, P., & Anastasiu, D. C. (2019, April). Stock price prediction using news sentiment analysis. In 2019 IEEE fifth international conference on big data computing service and applications (BigDataService) (pp. 205-208). IEEE.
[15] Chowdhury, S. G., Routh, S., & Chakrabarti, S. (2014). News analytics and sentiment analysis to predict stock price trends. International Journal of Computer Science and Information Technologies, 5(3), 3595-3604.
[16] Souma, W., Vodenska, I., & Aoyama, H. (2019). Enhanced news sentiment analysis using deep learning methods. Journal of Computational Social Science, 2(1), 33- 46.
[17] Costola, M., Hinz, O., Nofer, M., & Pelizzon, L. (2023). Machine learning sentiment analysis, COVID-19 news and stock market reactions. Research in International Business and Finance, 64, 101881.
[18] Bharathi, S., & Geetha, A. (2017). Sentiment analysis for effective stock market prediction. International Journal of Intelligent Engineering and Systems, 10(3), 146-154.
[19] Nemes, L., & Kiss, A. (2021). Prediction of stock values changes using sentiment analysis of stock news headlines. Journal of Information and Telecommunication, 5(3), 375-394.
[20] Januário, B. A., Carosia, A. E. D. O., da Silva, A. E. A., & Coelho, G. P. (2021). Sentiment analysis applied to news from the Brazilian stock market. IEEE Latin America Transactions, 20(3), 512-518.
[21] Mizumoto, K., Yanagimoto, H., & Yoshioka, M. (2012, May). Sentiment analysis of stock market news with semi- supervised learning. In 2012 IEEE/ACIS 11th International
Conference on Computer and Information Science (pp. 325- 328). IEEE.
[22] Agarwal, A. (2020, September). Sentiment analysis of financial news. In 2020 12th International Conference on Computational Intelligence and Communication Networks (CICN) (pp. 312-315). IEEE.
[23] Januário, B. A., Carosia, A. E. D. O., da Silva, A. E. A., & Coelho, G. P. (2021). Sentiment analysis applied to news from the Brazilian stock market. IEEE Latin America Transactions, 20(3), 512-518.
[24] Im Tan, L., San Phang, W., Chin, K. O., & Patricia, A. (2015, October). Rule-based sentiment analysis for financial news. In 2015 IEEE International Conference on Systems, Man, and Cybernetics (pp. 1601-1606). IEEE.
[25] Azar, P. D. (2009). Sentiment analysis in financial news (Doctoral dissertation, Harvard University).