CRAWL4_AI_BASE_DIRECTORY value , can't process AWS S3 folder link #523
Unanswered
keerthiganesh98
asked this question in
Forums - Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
As currently , i am trying to create a AWS Lambda function, where as base image i use public.ecr.aws/lambda/python:3.11 and install craw4ai package , once i test to run the aws lambda function by giving CRAWL4_AI_BASE_DIRECTORY = /tmp in AWS config , As aws lambda tmp folder have some duration of process to keep hold of data as far i know , because of which it fails of my execution by giving the error like
{
"errorMessage": "Connection closed while reading from the driver",
"errorType": "Exception",
"stackTrace": [
" File "/var/task/scrape_url.py", line 145, in lambda_handler\n return asyncio.run(async_lambda_handler(event, context))\n",
" File "/var/lang/lib/python3.11/site-packages/nest_asyncio.py", line 30, in run\n return loop.run_until_complete(task)\n",
" File "/var/lang/lib/python3.11/site-packages/nest_asyncio.py", line 98, in run_until_complete\n return f.result()\n",
" File "/var/lang/lib/python3.11/asyncio/futures.py", line 203, in result\n raise self._exception.with_traceback(self._exception_tb)\n",
" File "/var/lang/lib/python3.11/asyncio/tasks.py", line 277, in __step\n result = coro.send(None)\n",
" File "/var/task/scrape_url.py", line 155, in async_lambda_handler\n async with AsyncWebCrawler(verbose=True) as crawler:\n",
" File "/var/lang/lib/python3.11/site-packages/crawl4ai/async_webcrawler.py", line 218, in aenter\n return await self.start()\n",
" File "/var/lang/lib/python3.11/site-packages/crawl4ai/async_webcrawler.py", line 202, in start\n await self.crawler_strategy.aenter()\n",
" File "/var/lang/lib/python3.11/site-packages/crawl4ai/async_crawler_strategy.py", line 750, in aenter\n await self.start()\n",
" File "/var/lang/lib/python3.11/site-packages/crawl4ai/async_crawler_strategy.py", line 760, in start\n await self.browser_manager.start()\n",
" File "/var/lang/lib/python3.11/site-packages/crawl4ai/async_crawler_strategy.py", line 349, in start\n self.playwright = await async_playwright().start()\n",
" File "/var/lang/lib/python3.11/site-packages/playwright/async_api/_context_manager.py", line 51, in start\n return await self.aenter()\n",
" File "/var/lang/lib/python3.11/site-packages/playwright/async_api/_context_manager.py", line 46, in aenter\n playwright = AsyncPlaywright(next(iter(done)).result())\n",
" File "/var/lang/lib/python3.11/asyncio/futures.py", line 203, in result\n raise self._exception.with_traceback(self._exception_tb)\n"
]
}
Kindly pls give me some idea, how to overcome this !! Tx in advance
Beta Was this translation helpful? Give feedback.
All reactions