Repositories list

• ClueWeb22

  Public
  Python

  •1•13•1•0•Updated Dec 11, 2024Dec 11, 2024

• Lucindri

  Public
  Indri search implementation on top of Lucene search engine

  Java

  •5•34•3•6•Updated Mar 12, 2024Mar 12, 2024

• mturk

  Public
  Java

  •1•0•0•0•Updated Apr 14, 2023Apr 14, 2023

• CAsT-BERT-reranker

  Public
  Python

  •0•1•0•0•Updated Feb 2, 2021Feb 2, 2021

• clueweb12pp-core

  Public
  Unifies all code written for processing clueweb12++

  Clojure

  •0•2•0•0•Updated Aug 20, 2013Aug 20, 2013

• nopol

  Public
  Simple command line tool to export the ClueWeb dataset as HTML files.

  Java

  •5•5•1•0•Updated Jul 21, 2013Jul 21, 2013

• nabble-tools

  Public
  Tools for processing our nabble crawl

  Clojure

  •1•0•0•0•Updated May 13, 2013May 13, 2013

• ygroups-tools

  Public
  Processing the Yahoo groups crawl (part of clueweb12++)

  Clojure

  •0•0•0•0•Updated May 7, 2013May 7, 2013

• reddit_crawler

  Public
  Download the reddit dataset for ClueWeb12++

  Clojure

  •0•0•0•0•Updated May 4, 2013May 4, 2013

• scrapers

  Public
  Collection of Scrapers I am writing

  Clojure

  •0•0•0•0•Updated Apr 11, 2013Apr 11, 2013

• kba-tools

  Public
  Tools for processing the Trec KBA dataset

  Java

  •0•0•0•0•Updated Mar 16, 2013Mar 16, 2013

• warc-tools

  Public
  Command line utilities for working with WARC files

  Java

  •0•2•0•0•Updated Mar 13, 2013Mar 13, 2013

• crawling-tools

  Public
  A collection of tools for running the crawler, including scripts and scrapers to collect seeds.

  Python

  •0•0•0•0•Updated Feb 28, 2013Feb 28, 2013

• clueweb12pp

  Public
  Extension to the Clueweb12 dataset

  Python

  •0•3•0•0•Updated Feb 14, 2013Feb 14, 2013

• python-blekko

  Public
  bindings for the Blekko search engine API

  Python

  •2•0•0•0•Updated Feb 10, 2013Feb 10, 2013

• crawl-browser

  Public
  Scala

  •0•0•0•0•Updated Jan 31, 2013Jan 31, 2013

• oauth-racket

  Public
  Oauth Implementation in Racket (PLT Scheme)

  Racket

  •1•3•0•0•Updated Jan 7, 2013Jan 7, 2013

• heritrix-jobs

  Public
  Configuration files and scripts needed to run the Heritrix jobs

  1•1•0•0•Updated Dec 16, 2012Dec 16, 2012

• heritrix3

  Public
  Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project.

  Java

  •762•0•0•0•Updated Dec 8, 2012Dec 8, 2012

• language-id

  Public
  language identification module in racket

  Racket

  •0•0•0•0•Updated Dec 4, 2012Dec 4, 2012

• site-sampling

  Public
  Code to sample forums and check stats on pages and their information content

  Racket

  •0•0•0•0•Updated Nov 20, 2012Nov 20, 2012

• forum-seed

  Public
  Code and other stuff for discussion-forum seeds.

  Python

  •

  Other

  •1•1•1•0•Updated Oct 29, 2012Oct 29, 2012

• warc

  Public
  Python library for reading and writing warc files

  Python

  •

  GNU General Public License v2.0

  •114•1•0•0•Updated May 10, 2012May 10, 2012