[SPARKNLP-1092] Adding support to read HTML files

[danilojsl]

Description

This pull request introduces a new feature that enables reading and parsing HTML files into a structured Spark DataFrame. Leveraging this functionality allows for efficient processing and analysis of HTML content, seamlessly integrating with Spark NLP for enhanced downstream natural language processing tasks.

Key Changes

• Added sparknlp.read().html() Method: This method accepts file paths or URLs, parsing HTML content into a Spark DataFrame.
• Support for Varied Sources: The method is designed to handle both local directories, distributed file systems containing HTML files, and URLs, providing flexibility and broadening the range of possible data ingestion scenarios.

Motivation and Context

• Structured Data Representation: By transforming raw HTML content into a well-defined DataFrame structure, we enable seamless integration with Spark’s powerful analytical and data processing capabilities.
• Scalability: Leveraging Spark’s distributed architecture, this feature supports the efficient processing of large volumes of HTML data, which is critical for big data projects.
• Flexibility: The ability to handle both local files and URLs expands the scope of data sources that can be analyzed, facilitating diverse use cases in data engineering and analytics.
• Simplified Data Manipulation: Working with a structured DataFrame simplifies data manipulation tasks, such as filtering, aggregating, and transforming HTML content, reducing complexity and improving productivity.
• Enhanced Context for LLM Tasks: By organizing HTML data into structured formats, we can curate and provide more specific, context-rich content for large language model (LLM) prompts. This improves the quality of LLM-generated outputs by allowing for more targeted and relevant contextual information, which is essential for applications like NLU and content generation.

How Has This Been Tested?

Screenshots (if appropriate):

• Local Tests
• Google Colab notebook
• Databricks notebooks

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• Code improvements with no or little impact
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

• My code follows the code style of this project.
• My change requires a change to the documentation.
• I have updated the documentation accordingly.
• I have read the CONTRIBUTING page.
• I have added tests to cover my changes.
• All new and existing tests passed.