The PHP environment in ScraperWiki comes loaded with special functions in the namespace "scraperwiki::".
The source code implementation of these functions can be found here.
You can also use any PHP HTTP library.
- Returns the downloaded string
ScraperWiki provides a fully-fledged SQLite database for each scraper which you can save to. You can read the data back that has been committed by other scrapers, or extract it through the API.
- scraperwiki::save_sqlite($unique_keys, $data[, $table_name="swdata", $verbose=2])
- Saves a data record into the datastore into the table given by $table_name.
- $data is an array with field names as keys, $unique_keys must be an array that is a subset of array_keys($data) which determines when a record is over-written.
- For large numbers of records $data can be an array of arrays.
- verbose alters what is shown in the Data tab of the editor.
- scraperwiki::attach($name[, $asname])
Attaches to the datastore of another scraper named $name (which should be the short-name of the scraper, as it appears in the URL of its overview page).
$asname is an optional alias for the attached datastore.
- scraperwiki::select($val1[, $val2])
- Executes a select command on the datastore, e.g. select("* from swdata limit 10")
- Returns an array of key arrays for the records that have been selected.
- $val2 is an optional list of parameters if the select command contains '?'s.
- scraperwiki::sqliteexecute($val1[, $val2])
- Executes any arbitrary sqlite command (except attach), e.g. create, delete, insert or drop.
- $val2 is an optional indexed array of parameters if the command in $val1 contains question marks.
- (e.g. "insert into swdata values (?,?,?)").
- Commits to the file after a series of execute commands. (save_sqlite() auto-commits after every action).
- Returns an array of tables and their schemas in either the current or an attached database.
- Returns an array of attributes for each element of the table.
- scraperwiki::save_var($key, $value)
- Saves an arbitrary single-value into a sqlite table called "swvariables". e.g. Can be used to make scrapers able to continue after an interruption.
- scraperwiki::get_var($key[, $default])
- Retrieves a single value that was saved by save_var.
- Returns an array($lat, $lng) in WGS84 coordinates representing the central point of a UK postcode area.