Back to contents PHP Python Ruby Choose a language:

The Ruby environment in ScraperWiki comes with the ScraperWiki module loaded.

The source code implementation of these functions can be found in our bitbucket repository.

Scraping

You can also use any Ruby HTTP library, such as httpclient.

ScraperWiki::scrape(url[, params])
Returns the downloaded string from the given url.
params are sent using POST if set.

Datastore (SQLite)

ScraperWiki provides a fully-fledged SQLite database for each scraper which you can save to. You can read the data back that has been committed by other scrapers, or extract it through the API.

Datastore copy & paste guide for examples. SQL as understood by SQLite for the query language.

ScraperWiki::save_sqlite(unique_keys, data[, table_name="swdata", verbose=2])
Saves a data record into the datastore into the table given by table_name.
data is a hash with string or symbol field names as keys; unique_keys is an array that is a subset of data.keys which determines when a record is to be over-written.
For large numbers of records data can be a list of hashes.
verbose alters what is shown in the Data tab of the editor.
ScraperWiki::attach(name[, asname])

Attaches to the datastore of another scraper named name (which should be the short-name of the scraper, as it appears in the URL of its overview page).

asname is an optional alias for the attached datastore.

ScraperWiki::select(val1[, val2])
Executes a select command on the datastore, for example select("* from swdata limit 10").
Returns an array of hashes for the records that have been selected.
val2 is an optional array of parameters used when the select command contains '?'s.
ScraperWiki::sqliteexecute(cmd[, parms])
Executes any arbitrary sqlite command (except attach), for example create, delete, insert or drop.
parms is an optional array of parameters used when the command in cmd contains question marks.
(for example, "insert into swdata values (?,?,?)").
ScraperWiki::commit()
Commits to the file after a series of execute commands. (save_sqlite() auto-commits after every action).
ScraperWiki::show_tables([dbname])
Returns an array of tables and their schemas in either the current or an attached database.
ScraperWiki::table_info(name)
Returns an array of attributes for each element of the table.
ScraperWiki::save_var(key, value)
Saves an arbitrary single value into a sqlite table called "swvariables". Can be used to make scrapers able to continue after an interruption.
ScraperWiki::get_var(key[, default])
Retrieves a single value that was saved by save_var.

Views

ScraperWiki::httpresponseheader(headerkey, headervalue)
Set the content-type header to something other than HTML when using a ScraperWiki "view"
(for example "Content-Type", "image/png").

Geocoding

ScraperWiki::gb_postcode_to_latlng(postcode)
Returns an array [lat, lng] in WGS84 coordinates representing the central point of a UK postcode area.

Miscellaneous

require 'scrapers/name'
Imports the code from another scraper whose shortname is name (a scraper's shortname is what appears in the URL).

Exceptions

ScraperWiki::Error
This is the base class for all exceptions raised by the ScraperWiki library code. Currently there is only one subclass used (see below), but we like to leave room for future expansion.
ScraperWiki::CPUTimeExceededError

This is raised when a script running on ScraperWiki has used too much CPU time. This is implemented in a similar fashion across all our supported languages and is explained in a bit more detail in the FAQ.

This is a simple example of how to catch the exception.