Monterey scraping #166
Monterey scraping #166
Conversation
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
| print(hearings) | ||
| except Exception as e: | ||
| pass | ||
| return {"message": "Hello World!"} |
There was a problem hiding this comment.
Incomplete — haven't tested scrape_hearings()
| if upcoming_events_table: | ||
| events = upcoming_events_table.find_all("tr", class_="rgRow") | ||
|
|
||
| for event in events: |
There was a problem hiding this comment.
Incomplete/haven't tested. Still unsure about what fields can be extracted from the public hearings table, particularly for the Planning Committee hearings (which seem to be the only ones that have any useful information). So far, in the limited number of hearings that actually have relevant information, the fields that we can extract are:
- Hearing Date
- Location (sort of)
- Requesting Party (not explicit)
- APN
Missing (as far as I can see):
- County File Number
- Date Accepted
| @@ -9,7 +10,8 @@ export async function GET(req: Request) { | |||
|
|
|||
| try { | |||
| await connectDB(); | |||
| // TODO: get project by countyFileNumber | |||
| // TODO: get county from query params and use getProjectModel to get the correct model | |||
There was a problem hiding this comment.
For the base route that does a simple fetch all for the project data, since we now have multiple counties, we must separate the projects collection into 3 separate collections, one for each county. These will live under the respective slo, santaBarbara, and monterey databases. This GET request will have to take in an optional query parameter for the specified county in the request, to which it will map to the correct Mongo database, then query from its respective projects collection.
There was a problem hiding this comment.
NOTE: the query parameter is mostly required, but optional because we can make it so that if missing it will just choose the SLO County database as default
| @@ -19,8 +21,7 @@ export default function Navbar() { | |||
| <div className="flex bg-secondary px-16 h-16 items-center justify-between"> | |||
| <div> | |||
| <Link href="/" className="flex items-left"> | |||
| <div className="font-bold text-3xl">ECOLO</div> | |||
| <div className="text-primary text-3xl">GISTICS Web Scraper</div> | |||
| <Image src={Ecologistics} alt="Ecologistics Logo" /> | |||
There was a problem hiding this comment.
Adding the official logo to the navbar and replacing our makeshift one
| <div className="flex items-center py-4"> | ||
| <Select |
There was a problem hiding this comment.
Component that allows the user to select between the 3 different counties, to which their respective tables will be displayed
| console.log(columnToFilter); // log the columnToFilter state | ||
| }, [columnToFilter]); | ||
|
|
||
| const handleCountyChange = (selectedCounty: string) => { |
There was a problem hiding this comment.
Handler that will be primarily responsible for re-fetching the data (via the fetchProjectData function which should take in a new parameter county) that will populate the table for the user-selected county. Also responsible for any other state change, like the table header that displays the <County> (<number of projects>)
| @@ -48,6 +48,12 @@ export const projectSchema = new Schema<IProject>({ | |||
| additional_notes: { type: String, required: false, default: "N/A" }, | |||
| }); | |||
|
|
|||
| // retrieve correct collection based on county database | |||
There was a problem hiding this comment.
This will be the function used in the GET route to change the database according to the specified county in the request. Haven't tested it yet, but it looks reasonable as it uses the mongoose useDb function to select a database within the same mongo connection
Developer: Ryan Chan and Oli Lane
Closes #{ISSUE NUMBER HERE}
Pull Request Summary
TBD
Modifications
{list out the files created/modified and a brief description of what was changed}
Testing Considerations
{list out what you have tested and what the reviewer should verify}
Pull Request Checklist
Screenshots/Screencast
{put screenshots of your change, or even better a screencast displaying the functionality}