American Assiociation of Individual Investors (AAII) Sentiment over Time

This chart visualizes investor sentiment over time using data from the American Association of Individual Investors (AAII). The dataset, which is community-contributed, offers insights into how investors feel—whether bullish, bearish, or neutral—on a weekly basis. My aim was to explore whether there was a consensus among groups of investors and how these sentiments evolved over time. Cleaning this dataset was a significant challenge, as it was messy and required extensive preprocessing to make it suitable for graphing. Despite these hurdles, I transformed the raw data into a clean, structured format, enabling it to be visualized effectively. The chart highlights key moments in market history, such as the 2008 financial crisis, where bearish sentiment spiked significantly. However, no single sentiment category dominates over the long term, reflecting the diversity of perspectives within the investor community. The visualization includes an interactive slider, allowing users to explore the dataset at both macro (long-term) and micro (short-term) levels, making it adaptable for analyzing specific periods or trends. This time series analysis underscores the importance of tracking collective investor sentiment during different economic conditions. By combining historical trends with interactive exploration, this visualization provides valuable insights into the dynamics of investor psychology across time.

References:

• Sentiment Survey Historical Data
• AAIISentimentCleaner.ipynb
• Cleaned_AAIIsentiment.csv

Scatter Correlation & Regression: VIX and Fear and Greed?

This chart explores the correlation between the CNN Fear & Greed Index and the VIX, a widely recognized measure of market volatility. The Fear & Greed Index was my chosen dataset for exploring sentiment, ranging from 0 (extreme fear) to 100 (extreme greed). The VIX measures market expectations of future volatility. I aimed to explore the relationship between these metrics. Correlating the data in Colab using Matplotlib showed a correlation coefficient of -0.445, meaning that as fear increased, volatility tended to rise. My initial hypothesis was that heightened fear leads to irrational behavior, resulting in greater market volatility. I also implemented a 30-day rolling correlation to examine how the relationship evolved over time, revealing variations in the strength and direction of the correlation during different market conditions. The chart features a scatter plot to illustrate the overall correlation, complemented by a regression line for clarity. Additionally, a time series of the rolling correlation highlights dynamic changes in the relationship, particularly during key economic events.

References:

• Historical Data for Cboe VIX® Index
• VIX_History.csv
• fear-greed-2011-2023.csv
• CC8&10 AA and ML .ipynb

VIX and Fear & Greed Histograms

Correlation Sentiment and Returns

This section explores the relationship between investor sentiment and market returns. The correlation coefficient between sentiment, as measured by the Fear & Greed Index, and market returns was calculated to be 0.34, indicating a weak positive relationship. This suggests that higher sentiment values (indicating greed) are slightly associated with improved market performance, while lower sentiment values (indicating fear) may correspond to weaker returns. To analyze this further, I used Colab to run the regression and calculate the correlation coefficient through a heatmap visualization. The heatmap allowed me to explore the strength of the relationship across multiple return metrics, providing a clearer picture of how sentiment influences market behavior. I utilized the datasets I had previously cleaned and prepared for this analysis, enabling an efficient and streamlined process. These results provide insights into the subtle yet impactful dynamics between market sentiment and financial performance

References:

• SENT&Returns.ipynb
• Cleaned_AAIIsentiment.csv

Clustering Analysis: VIX, Returns, and Fear & Greed

To understand market conditions, I grouped the indexes—Fear & Greed, VIX, and market returns—using a clustering approach. This process was implemented in Colab with the K-Means algorithm, which identifies distinct groups within the data by minimizing intra-cluster variation. Before applying K-Means, I preprocessed the data by cleaning, normalizing, and scaling the variables to ensure accurate clustering results. The analysis revealed meaningful patterns within the clusters. Cluster 0 represented a "bullish market," characterized by high greed, low volatility, and strong positive returns. On the other hand, Cluster 2 was identified as a "bearish market," marked by high fear, high volatility, and negative returns. A neutral cluster reflected transitional conditions with moderate sentiment and volatility. Through the clustering process, I was able to group relationships between sentiment, volatility, and returns under different market conditions. This gave me a perspective on investor behavior, allowing me to transform complex data into a clear understanding of how these factors interact in financial markets.

To analyze this relationship further, I implemented a Random Forest regression model. This algorithm constructs multiple decision trees and combines their predictions to enhance accuracy and robustness. Using scikit-learn in Colab, I processed the data, trained the model, and evaluated its performance. The results indicated that other factors also influence the relationship between sentiment, volatility, and market returns, as reflected by the model's performance metrics. Further analysis of feature importance revealed that the VIX (volatility) was the most significant variable in predicting returns, with a contribution of 0.6522 compared to sentiment as measured by the Fear & Greed Index (0.3478). This finding highlights the dominant role of market volatility in influencing financial outcomes, though sentiment remains a contributing factor. Through this process, I was able to gain deeper insights into the dynamics of these variables, providing a foundation for understanding how external factors shape market behavior.

References:

• CC8&10 AA and ML .ipynb
• SENT&Returns.ipynb
• Cleaned_AAIIsentiment.csv
• Historical Data for Cboe VIX® Index
• VIX_History.csv
• fear-greed-2011-2023.csv