핵심 요구사항
- Wayback Machine 스냅샷에서 텍스트·이미지·메타데이터를 개별 셀렉터로 추출
- 오프라인에서 열리는 자체 완결형 페이지 저장 및 서브페이지 크롤링 옵션
- 동일 정보를 담는 병렬 JSON 파일 생성
- URL 리스트·날짜 범위·출력 디렉터리를 받는 CLI 구현
- 대량 배치의 차단/타임아웃 방지(속도·복원력) 및 에러 로깅·요약 리포트
- 전체 타임라인에 걸친 수집과 재현 가능한 환경 구성
Required stack
필요 기술
PythonBeautifulSoupScrapyrequestsNode.jsPuppeteerJSON
Project brief
프로젝트 내용
I have a personal list of several thousand archived URLs on the Wayback Machine and need a repeatable workflow that can process them in bulk. For every snapshot the workflow should:
• Pull out the full text content, all embedded images, and any available metadata (titles, descriptions, dates, alt tags, etc.). Give separate selectors so I'm able to extract only the required data.
• Save a clean, self-contained copy of the page so it can be opened offline as a standalone webpage. A option to go discover for subpages and extract data from them too.
• Generate a parallel JSON file that structures the same information in a machine-readable way.
Speed and resilience matter; the run must cope with large batches without timing out or getting blocked by the Wayback servers. If an individual URL fails, the process should log the error and continue, with a summary report at the end. I need them to extract webpages across all time, in a frequency I request for.
Please build this in an easily reproducible environment—Python with requests/BeautifulSoup or Scrapy, Node with Puppeteer, or another language you can justify—and document any dependencies so I can spin it up on my own machine. A simple command-line interface that accepts my URL list, an optional date range, and an output directory is ideal. It has to work across the whole timeline.
Deliverables
1. Source code and any helper scripts.
2. Instructions to install and run (readme or screencast).
3. A short test run on 200 sample URLs demonstrating the JSON output and offline pages.
4. Error-handling and logging strategy.
I’ll consider the job complete once I can replicate your results on my side with no manual tweaks.
I'll share the 200 urls that need to be tested.
• Pull out the full text content, all embedded images, and any available metadata (titles, descriptions, dates, alt tags, etc.). Give separate selectors so I'm able to extract only the required data.
• Save a clean, self-contained copy of the page so it can be opened offline as a standalone webpage. A option to go discover for subpages and extract data from them too.
• Generate a parallel JSON file that structures the same information in a machine-readable way.
Speed and resilience matter; the run must cope with large batches without timing out or getting blocked by the Wayback servers. If an individual URL fails, the process should log the error and continue, with a summary report at the end. I need them to extract webpages across all time, in a frequency I request for.
Please build this in an easily reproducible environment—Python with requests/BeautifulSoup or Scrapy, Node with Puppeteer, or another language you can justify—and document any dependencies so I can spin it up on my own machine. A simple command-line interface that accepts my URL list, an optional date range, and an output directory is ideal. It has to work across the whole timeline.
Deliverables
1. Source code and any helper scripts.
2. Instructions to install and run (readme or screencast).
3. A short test run on 200 sample URLs demonstrating the JSON output and offline pages.
4. Error-handling and logging strategy.
I’ll consider the job complete once I can replicate your results on my side with no manual tweaks.
I'll share the 200 urls that need to be tested.
지원 기회 분석
지원 전에 볼 것
예상 산출물
- 소스 코드 및 헬퍼 스크립트
- 설치/실행 문서(README 또는 스크린캐스트)
- 200개 샘플 URL 대상 테스트 실행 결과(JSON 출력 + 오프라인 페이지)
- 에러 처리 및 로깅 전략 문서
매력 포인트
- 요구사항과 산출물이 명확하게 정의됨
- 기술 스택 선택의 자유도가 높음
- 200개 샘플로 검증 가능한 명확한 완료 기준
확인할 점
- 입찰 184건으로 경쟁이 매우 치열함
- '전체 타임라인'·'수천 URL' 등 범위가 넓은데 예산($100~800 SGD)이 상대적으로 낮음
- 클라이언트 정보 공백(국가·고용주 ID 없음)으로 신뢰도 확인 어려움
Client signal
클라이언트 정보
플랫폼에서 상세 정보를 확인하세요.