Back to contents PHP Python Ruby Choose a language:

The PHP environment in ScraperWiki comes loaded with special functions in the namespace "scraperwiki::".

The source code implementation of these functions can be found here.

Scraping

You can also use any PHP HTTP library.

scraperwiki::scrape($url)
Returns the downloaded string

Datastore (SQLite)

ScraperWiki provides a fully-fledged SQLite database for each scraper which you can save to. You can read the data back that has been committed by other scrapers, or extract it through the API.

Datastore copy & paste guide for examples. SQL as understood by SQLite for the query language.

scraperwiki::save_sqlite($unique_keys, $data[, $table_name="swdata", $verbose=2])
Saves a data record into the datastore into the table given by $table_name.
$data is an array with field names as keys, $unique_keys must be an array that is a subset of array_keys($data) which determines when a record is over-written.
For large numbers of records $data can be an array of arrays.
verbose alters what is shown in the Data tab of the editor.
scraperwiki::attach($name[, $asname])

Attaches to the datastore of another scraper named $name (which should be the short-name of the scraper, as it appears in the URL of its overview page).

$asname is an optional alias for the attached datastore.

scraperwiki::select($val1[, $val2])
Executes a select command on the datastore, e.g. select("* from swdata limit 10")
Returns an array of key arrays for the records that have been selected.
$val2 is an optional list of parameters if the select command contains '?'s.
scraperwiki::sqliteexecute($val1[, $val2])
Executes any arbitrary sqlite command (except attach), e.g. create, delete, insert or drop.
$val2 is an optional indexed array of parameters if the command in $val1 contains question marks.
(e.g. "insert into swdata values (?,?,?)").
scraperwiki::sqlitecommit()
Commits to the file after a series of execute commands. (save_sqlite() auto-commits after every action).
scraperwiki::show_tables([$dbname])
Returns an array of tables and their schemas in either the current or an attached database.
scraperwiki::table_info($name)
Returns an array of attributes for each element of the table.
scraperwiki::save_var($key, $value)
Saves an arbitrary single-value into a sqlite table called "swvariables". e.g. Can be used to make scrapers able to continue after an interruption.
scraperwiki::get_var($key[, $default])
Retrieves a single value that was saved by save_var.

Views

scraperwiki::httpresponseheader($headerkey, $headervalue)
Set the content-type header to something other than HTML when using a ScraperWiki "view"
(e.g. "Content-Type", "image/png")

Geocoding

scraperwiki::gb_postcode_to_latlng($postcode)
Returns an array($lat, $lng) in WGS84 coordinates representing the central point of a UK postcode area.