More

CartoDB import defaults to string

CartoDB import defaults to string


When uploading CSV files to CartoDB through their web interface, all columns default to string as their data type, despite the CSV file containing a number of number-only columns. An example row is

1400000US55025000100,55025000100,"Census Tract 1, Dane County, Wisconsin",1401,118,1070,128,903,136,167,82,131,62,36,34,0,9,179,74,179,74,0,9,0,9,0,9,0,9,36,31,35,28,6,11,75,45

I know I can change the data type in the table editor, but when you have a large number of columns, this becomes very tedious. According to their documentation, CartoDB should automatically assigns the right data type. Any advice on how to make this work?


CSV does not have the concept of data types and that's why CartoDB tries to guess the type of the column. In any case CartoDB can not have a 100% success rate.

if you want to ensure your data types are kept use a format that allow data types (geojson, shp for example)

In this case I think it's a problem with ogr2ogr (the tool we use under the hood to process files). I reported this as a bug https://github.com/CartoDB/cartodb/issues/2600 so you can track it


Chapter 4

public static void ifElseMystery1(int x, int y) <
int z = 4
if (z <= x) <
z = x + 1
> else <
z = z + 9
>
if (z <= y) <
y++
>
System.out.println(z + " " + y)
>
For each call below, indicate what output is produced.
ifElseMystery1(3, 20)

public static void ifElseMystery2(int a, int b) <
if (a * 2 < b) <
a = a * 3
>
if (b < a) <
b++
> else <
a--
>
System.out.println(a + " " + b)
>
For each call below, indicate what output is produced.
ifElseMystery2(10, 2)

System.out.print("Is your money multiplied 1 or 2 times? ")
int times = console.nextInt()

Here are two sample executions:

(The idea here is that the expression 3.2 * 3 in Java does not exactly equal 9.6. You should leave in the computation of the credits variable and should still make your test examine that variable's value, rather than just directly testing the gpa variable itself.)

int x = 4
int y = -3
int z = 4
What are the results of the following relational expressions?

public class CharMystery <
public static void printRange(char startLetter, char endLetter) <
for (char letter = startLetter letter <= endLetter letter++) <
System.out.print(letter)
>
System.out.println()
>

In other words, the method should generate the following sequence:

printRange(2, 7)
printRange(19, 11)
printRange(5, 5)
The output produced should be the following:

The following table lists some calls to your method and their expected output:

>else if (cols==size-rows+1) <
System.out.print("x")

>
System.out.println("Smallest = "+s)
System.out.println("Largest = "+a)

The longest name should be printed with its first letter capitalized and all subsequent letters in lowercase, regardless of the capitalization the user used when typing in the name. If there is a tie for longest between two or more names, use the tied name that was typed earliest. Also print a message saying that there was a tie, as in the right log below. It's possible that some shorter names will tie in length, such as DANE and Erik in the left log below but don't print a message unless the tie is between the longest names. You may assume that n is at least 1, that each name is at least 1 character long, and that the user will type single-word names consisting of only letters. The following table shows two sample calls and their output:


  • Visualize spatial data programmatically as matplotlib images or embedded interactive maps
  • Perform cloud-based spatial data processing using CARTO’s analysis tools
  • Extract, transform, and Load (ETL) data using the Python ecosystem for getting data into and out of CARTO
  • Data Services integrations using CARTO’s Data Observatory and other Data Services APIs

The easiest way to try out cartoframes is to use the cartoframes example notebooks running in binder: https://mybinder.org/v2/gh/CartoDB/cartoframes/master?filepath=examples If you already have an API key, you can follow along and complete all of the example notebooks.

If you do not have an API key, you can use the Example Context to read the example data, make maps, and run arbitrary queries from the datasets there. The best place to get started is in the “Example Datasets” notebook found when running binder or downloading from the examples directory in the cartoframes GitHub repository.

The example context only provides read access, so not all cartoframes features are available. For full access, Start a free 30 day trial or get free access with a GitHub Student Developer Pack.


Cartoframes.context module¶

CartoContext class for authentication with CARTO and high-level operations such as reading tables from CARTO into dataframes, writing dataframes to CARTO tables, creating custom maps from dataframes and CARTO tables, and augmenting data using CARTO’s Data Observatory. Future methods will interact with CARTO’s services like routing, geocoding, and isolines, PostGIS backend for spatial processing, and much more.

Manages connections with CARTO for data and map operations. Modeled after SparkContext.

There are two ways of authenticating against a CARTO account:

  1. Setting the base_url and api_key directly in CartoContext . This method is easier.:

  • base_url (str) – Base URL of CARTO user account. Cloud-based accounts should use the form https://.carto.com (e.g., https://eschbacher.carto.com for user eschbacher ) whether on a personal or multi-user account. On-premises installation users should ask their admin.
  • api_key (str) – CARTO API key.
  • creds ( Credentials ) – A Credentials instance can be used in place of a base_url / api_key combination.
  • session (requests.Session,optional) – requests session. See requests documentation for more information.
  • verbose (bool,optional) – Output underlying process states (True), or suppress (False, default)

A CartoContext object that is authenticated against the user’s CARTO account.

Create a CartoContext object for a cloud-based CARTO account.

If using cartoframes with an on premises CARTO installation, sometimes it is necessary to disable SSL verification depending on your system configuration. You can do this using a requests Session object as follows:

Get an augmented CARTO dataset with Data Observatory measures. Use CartoContext.data_discovery to search for available measures, or see the full Data Observatory catalog. Optionally persist the data as a new table.

Get a DataFrame with Data Observatory measures based on the geometries in a CARTO table.

Pass in cherry-picked measures from the Data Observatory catalog. The rest of the metadata will be filled in, but it’s important to specify the geographic level as this will not show up in the column name.

  • table_name (str) – Name of table on CARTO account that Data Observatory measures are to be added to.
  • metadata (pandas.DataFrame) – List of all measures to add to table_name . See CartoContext.data_discovery outputs for a full list of metadata columns.
  • persist_as (str,optional) – Output the results of augmenting table_name to persist_as as a persistent table on CARTO. Defaults to None , which will not create a table.
  • how (str,optional) – Not fully implemented. Column name for identifying the geometry from which to fetch the data. Defaults to the_geom , which results in measures that are spatially interpolated (e.g., a neighborhood boundary’s population will be calculated from underlying census tracts). Specifying a column that has the geometry identifier (for example, GEOID for US Census boundaries), results in measures directly from the Census for that GEOID but normalized how it is specified in the metadata.

A DataFrame representation of table_name which has new columns for each measure in metadata .

  • NameError – If the columns in table_name are in the suggested_name column of metadata .
  • ValueError – If metadata object is invalid or empty, or if the number of requested measures exceeds 50.
  • CartoException – If user account consumes all of Data Observatory quota

DEPRECATED. Use CartoContext.data instead

data_boundaries ( boundary=None, region=None, decode_geom=False, timespan=None, include_nonclipped=False ) ¶

Find all boundaries available for the world or a region . If boundary is specified, get all available boundary polygons for the region specified (if any). This method is espeically useful for getting boundaries for a region and, with CartoContext.data and CartoContext.data_discovery , getting tables of geometries and the corresponding raw measures. For example, if you want to analyze how median income has changed in a region (see examples section for more).

Find all boundaries available for Australia. The columns geom_name gives us the name of the boundary and geom_id is what we need for the boundary argument.

Get the boundaries for Australian Postal Areas and map them.

Get census tracts around Idaho Falls, Idaho, USA, and add median income from the US census. Without limiting the metadata, we get median income measures for each census in the Data Observatory.

Region where boundary information or, if boundary is specified, boundary polygons are of interest. region can be one of the following:

  • table name (str): Name of a table in user’s CARTO account
  • bounding box (list of float): List of four values (two lng/lat pairs) in the following order: western longitude, southern latitude, eastern longitude, and northern latitude. For example, Switzerland fits in [5.9559111595,45.8179931641,10.4920501709,47.808380127]

If boundary is specified, then all available boundaries and accompanying geom_refs in region (or the world if region is None or not specified) are returned. If boundary is not specified, then a DataFrame of all available boundaries in region (or the world if region is None )

Discover Data Observatory measures. This method returns the full Data Observatory metadata model for each measure or measures that match the conditions from the inputs. The full metadata in each row uniquely defines a measure based on the timespan, geographic resolution, and normalization (if any). Read more about the metadata response in Data Observatory documentation.

Internally, this method finds all measures in region that match the conditions set in keywords , regex , time , and boundaries (if any of them are specified). Then, if boundaries is not specified, a geographical resolution for that measure will be chosen subject to the type of region specified:

  1. If region is a table name, then a geographical resolution that is roughly equal to region size / number of subunits .
  2. If region is a country name or bounding box, then a geographical resolution will be chosen roughly equal to region size / 500 .

Since different measures are in some geographic resolutions and not others, different geographical resolutions for different measures are oftentimes returned.

To remove the guesswork in how geographical resolutions are selected, specify one or more boundaries in boundaries . See the boundaries section for each region in the Data Observatory catalog.

The metadata returned from this method can then be used to create raw tables or for augmenting an existing table from these measures using CartoContext.data . For the full Data Observatory catalog, visit https://cartodb.github.io/bigmetadata/. When working with the metadata DataFrame returned from this method, be careful to only remove rows not columns as CartoContext.data <cartoframes.context.CartoContext.data> generally needs the full metadata.

Narrowing down a discovery query using the keywords , regex , and time filters is important for getting a manageable metadata set. Besides there being a large number of measures in the DO, a metadata response has acceptable combinations of measures with demonimators (normalization and density), and the same measure from other years.

For example, setting the region to be United States counties with no filter values set will result in many thousands of measures.

Get all European Union measures that mention freight .

Information about the region of interest. region can be one of three types:

  • region name (str): Name of region of interest. Acceptable values are limited to: ‘Australia’, ‘Brazil’, ‘Canada’, ‘European Union’, ‘France’, ‘Mexico’, ‘Spain’, ‘United Kingdom’, ‘United States’.
  • table name (str): Name of a table in user’s CARTO account with geometries. The region will be the bounding box of the table.

Note

If a table name is also a valid Data Observatory region name, the Data Observatory name will be chosen over the table.

Note

Geometry levels are generally chosen by subdividing the region into the next smallest administrative unit. To override this behavior, specify the boundaries flag. For example, set boundaries to 'us.census.tiger.census_tract' to choose US census tracts.

A dataframe of the complete metadata model for specific measures based on the search parameters.

  • ValueError – If region is a list and does not consist of four elements, or if region is not an acceptable region
  • CartoException – If region is not a table in user account

Delete a table in user’s CARTO account.

Parameters:table_name (str) – Name of table to delete
Returns:None
map ( layers=None, interactive=True, zoom=None, lat=None, lng=None, size=(800, 400), ax=None ) ¶

Produce a CARTO map visualizing data layers.

Create a map with two data Layer s, and one BaseMap layer:

Create a snapshot of a map at a specific zoom and center:

List of zero or more of the following:

    : cartoframes Layer object for visualizing data from a CARTO table. See Layer for all styling options. : Basemap for contextualizng data layers. See BaseMap for all styling options. : Layer from an arbitrary query. See QueryLayer for all styling options.

Interactive maps are rendered as HTML in an iframe , while static maps are returned as matplotlib Axes objects or IPython Image.

IPython.display.HTML or matplotlib Axes

Pull the result from an arbitrary SQL query from a CARTO account into a pandas DataFrame. Can also be used to perform database operations (creating/dropping tables, adding columns, updates, etc.).

  • query (str) – Query to run against CARTO user database. This data will then be converted into a pandas DataFrame.
  • table_name (str,optional) – If set, this will create a new table in the user’s CARTO account that is the result of the query. Defaults to None (no table created).
  • decode_geom (bool,optional) – Decodes CARTO’s geometries into a Shapely object that can be used, for example, in GeoPandas.

DataFrame representation of query supplied. Pandas data types are inferred from PostgreSQL data types. In the case of PostgreSQL date types, dates are attempted to be converted, but on failure a data type ‘object’ is used.

Query a table in CARTO and write a new table that is result of query. This query gets the 10 highest values from a table and returns a dataframe, as well as creating a new table called ‘top_ten’ in the CARTO account.

This query joins points to polygons based on intersection, and aggregates by summing the values of the points in each polygon. The query returns a dataframe, with a geometry column that contains polygons and also creates a new table called ‘points_aggregated_to_polygons’ in the CARTO account.

Read a table from CARTO into a pandas DataFrames.

  • table_name (str) – Name of table in user’s CARTO account.
  • limit (int,optional) – Read only limit lines from table_name . Defaults to None , which reads the full table.
  • index (str,optional) – Not currently in use.
  • decode_geom (bool,optional) – Decodes CARTO’s geometries into a Shapely object that can be used, for example, in GeoPandas.
  • shared_user (str,optional) – If a table has been shared with you, specify the user name (schema) who shared it.

DataFrame representation of table_name from CARTO.

Depending on the size of the DataFrame or CARTO table, perform granular operations on a DataFrame to only update the changed cells instead of a bulk upload. If on the large side, perform granular operations, if on the smaller side use Import API.

List all tables in user’s CARTO account

Returns: list of Table
write ( df, table_name, temp_dir='/home/docs/.cache/cartoframes', overwrite=False, lnglat=None, encode_geom=False, geom_col=None, **kwargs ) ¶

Write a DataFrame to a CARTO table.

Write a pandas DataFrame to CARTO.

Scrape an HTML table from Wikipedia and send to CARTO with content guessing to create a geometry from the country column. This uses a CARTO Import API param content_guessing parameter.

Keyword arguments to control write operations. Options are:

  • compression to set compression for files sent to CARTO. This will cause write speedups depending on the dataset. Options are None (no compression, default) or gzip .
  • Some arguments from CARTO’s Import API. See the params listed in the documentation for more information. For example, when using content_guessing=’true’ , a column named ‘countries’ with country names will be used to generate polygons for each country. Another use is setting the privacy of a dataset. To avoid unintended consequences, avoid file , url , and other similar arguments.

BatchJobStatus or None: If lnglat flag is set and the DataFrame has more than 100,000 rows, a BatchJobStatus instance is returned. Otherwise, None.

DataFrame indexes are changed to ordinary columns. CARTO creates an index called cartodb_id for every table that runs from 1 to the length of the DataFrame.


CartoDB import defaults to string - Geographic Information Systems

Tagged Image File Format, abbreviated TIFF or TIF, is a computer file format for storing raster graphics images, popular among graphic artists, the publishing industry,[1] and photographers. TIFF is widely supported by scanning, faxing, word processing, optical character recognition, image manipulation, desktop publishing, and page-layout applications.[2] The format was created by Aldus Corporation for use in desktop publishing. It published the latest version 6.0 in 1992, subsequently updated with an Adobe Systems copyright after the latter acquired Aldus in 1994.

The GDAL lib could decode tif file with full geography information.

also could read file in anywhere:

fast_percentager_strentching(image=None, percentage=2, sample=10000)

the huge imagery percentage strentching is a slow process , this project re-design the strentching algorithm to boost the process to o(1).

eg - output (original percentage strentching algorithm (30s) )

eg - ouput (fast percentage strentching algorithm (1s))

if the image is none , default image is result of Class.image

write a pure image file like png

Some raster has wrong size or GDAL datasets.This function will keep the geography information & resize image .

vectorize binary ndarray . you must set binary uint8 ndarray data first. That will generate shapefile of vectorized binary information.

All the transform function could work that base on you already read a tiff file that has decoded to gdal dataset successful. like :

that will transform the image cord to geo cord.

clear all the class status of IO

this class already support some vector file format:

Additionally, the Vector class inherit the Raster class , so you could use and call all function with single instance.like:

you could use this class to solve some rasterize problems

Choose layer name as default layer. eg:

get vector file bounding box cord

SaveVectorByLayerName(LayerName, outputname, format="GeoJSON") This function could be save the one layer of multi-layer vector object .

Rasterize(outputname, Nodata=0) map the vector layer with raster dataset and generate rasterized vector file. Tips : this function just read defaultlayer ,so must set default layer first.

reset the default layer & all layer option

downloader is resources lib & all the source could found here like :

I don't test the limit of download count . But different country may be has totally different problem. For example ,in china internet enviroment ,'Google Satellite' cloud be nothing response with code 404. At this time, we use the OSM standard to support datasource list:

  • Google
  • Google China,
  • Google Maps,
  • Google Satellite,
  • Google Terrain,
  • Google Terrain Hybrid,
  • Google Satellite Hybrid
  • Stamen Terrain
  • Stamen Toner
  • Stamen Toner Light
  • Stamen Watercolor
  • Wikimedia Map
  • Wikimedia Hike Bike Map
  • Esri Boundaries Places
  • Esri Gray (dark)
  • Esri Gray (light)
  • Esri National Geographic
  • Esri Ocean,
  • Esri Satellite,
  • Esri Standard,
  • Esri Terrain,
  • Esri Transportation,
  • Esri Topo World,
  • OpenStreetMap Standard,
  • OpenStreetMap H.O.T.,
  • OpenStreetMap Monochrome,
  • OpenTopoMap,
  • Strava All,
  • Strava Run,
  • Open Weather Map Temperature,
  • Open Weather Map Clouds,
  • Open Weather Map Wind Speed,
  • CartoDb Dark Matter,
  • CartoDb Positron,
  • Bing VirtualEarth

add_cord(self, left, top, right, bottom, zoom, style='s'):

Compute the Extent by (x1,y1,x2,y2) -Retangle coordinate A

addcord() as a function ,input is WGS cord of left-top point & right-bottom point x1,y1,x2,y2,additional zoom level that mean different level density of data grid.

download(output_path="./images") download the file to output_path

Merge the all file to single file or not

vector : local path of vector object

cord : WGS Standard Cord (min-lon,min-lat,max-lon,maxlat,zoom_level)

class_key : The class you need generate

DataSourcesType : Map production datasource name

Merge : Merge the tiles file to whole file or not

Keep_local : The last step will delete local cache ,this option could choose to save it.

Upload : Use Network Strong Server or not (Just support Huawei OBS server now)

Thread_count : download thread count

Over_write : overwrite remote files or not

QGIS Map Resources Script:

Copyright 2020 Tom Winshare

Licensed under the Apache License, Version 2.0 (the "License") you may not use this file except in compliance with the License. You may obtain a copy of the License at

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.


MATCH predicate¶

The MATCH predicate can be used to perform multiple kinds of searches on indices or indexed columns. While it can be used to perform fulltext searches on analyzed indices of type text , it is also handy for operating on geographic indices, querying for relations between geographical shapes and points.

The MATCH predicate for geographical search supports a single column_ident of a geo_shape indexed column as first argument.

The second argument, the query_term is taken to match against the indexed geo_shape .

The matching operation is determined by the match_type which determines the spatial relation we want to match. Available match_types are:

(Default) If the two shapes share some points and/or area, they are intersecting and considered matching using this match_type . This also precludes containment or complete equality.

If the two shapes share no single point or area, they are disjoint. This is the opposite of intersects .

If the indexed column_ident shape is completely inside the query_term shape, they are considered matching using this match_type .

The MATCH predicate can only be used in the WHERE clause and on user-created tables. Using the MATCH predicate on system tables is not supported.

One MATCH predicate cannot combine columns of both relations of a join.

Additionally, MATCH predicates cannot be used on columns of both relations of a join if they cannot be logically applied to each of them separately. For example:

Having a table countries with a GEO_SHAPE column geo , indexed using geohash , you can query that column using the MATCH predicate with different match types as described above:


CartoContext¶

CartoContext class for authentication with CARTO and high-level operations such as reading tables from CARTO into dataframes, writing dataframes to CARTO tables, creating custom maps from dataframes and CARTO tables, and augmenting data using CARTO’s Data Observatory. Future methods will interact with CARTO’s services like routing, geocoding, and isolines, PostGIS backend for spatial processing, and much more.

Manages connections with CARTO for data and map operations. Modeled after SparkContext.

There are two ways of authenticating against a CARTO account:

  1. Setting the base_url and api_key directly in CartoContext . This method is easier.:

  • base_url (str) – Base URL of CARTO user account. Cloud-based accounts should use the form https://.carto.com (e.g., https://eschbacher.carto.com for user eschbacher ) whether on a personal or multi-user account. On-premises installation users should ask their admin.
  • api_key (str) – CARTO API key.
  • creds ( Credentials ) – A Credentials instance can be used in place of a base_url / api_key combination.
  • session (requests.Session,optional) – requests session. See requests documentation for more information.
  • verbose (bool,optional) – Output underlying process states (True), or suppress (False, default)

A CartoContext object that is authenticated against the user’s CARTO account.

Create a CartoContext object for a cloud-based CARTO account.

If using cartoframes with an on premises CARTO installation, sometimes it is necessary to disable SSL verification depending on your system configuration. You can do this using a requests Session object as follows:

Write a DataFrame to a CARTO table.

Write a pandas DataFrame to CARTO.

Scrape an HTML table from Wikipedia and send to CARTO with content guessing to create a geometry from the country column. This uses a CARTO Import API param content_guessing parameter.

datetime64[ns] column will lose precision sending a dataframe to CARTO because postgresql has millisecond resolution while pandas does nanoseconds

Keyword arguments to control write operations. Options are:

  • compression to set compression for files sent to CARTO. This will cause write speedups depending on the dataset. Options are None (no compression, default) or gzip .
  • Some arguments from CARTO’s Import API. See the params listed in the documentation for more information. For example, when using content_guessing=’true’ , a column named ‘countries’ with country names will be used to generate polygons for each country. Another use is setting the privacy of a dataset. To avoid unintended consequences, avoid file , url , and other similar arguments.

DataFrame indexes are changed to ordinary columns. CARTO creates an index called cartodb_id for every table that runs from 1 to the length of the DataFrame.

List all tables in user’s CARTO account

  • table_name (str) – Name of table in user’s CARTO account.
  • limit (int,optional) – Read only limit lines from table_name . Defaults to None , which reads the full table.
  • decode_geom (bool,optional) – Decodes CARTO’s geometries into a Shapely object that can be used, for example, in GeoPandas.
  • shared_user (str,optional) – If a table has been shared with you, specify the user name (schema) who shared it.
  • retry_times (int,optional) – If the read call is rate limited, number of retries to be made

DataFrame representation of table_name from CARTO.

Delete a table in user’s CARTO account.

Parameters:table_name (str) – Name of table to delete
Returns: True if table is removed
Return type:bool
query ( query, table_name=None, decode_geom=False, is_select=None )

Pull the result from an arbitrary SQL SELECT query from a CARTO account into a pandas DataFrame. This is the default behavior, when is_select=True

Can also be used to perform database operations (creating/dropping tables, adding columns, updates, etc.). In this case, you have to explicitly specify is_select=False

This method is a helper for the CartoContext.fetch and CartoContext.execute methods. We strongly encourage you to use any of those methods depending on the type of query you want to run. If you want to get the results of a SELECT query into a pandas DataFrame, then use CartoContext.fetch . For any other query that performs an operation into the CARTO database, use CartoContext.execute

  • query (str) – Query to run against CARTO user database. This data will then be converted into a pandas DataFrame.
  • table_name (str,optional) – If set (and is_select=True ), this will create a new table in the user’s CARTO account that is the result of the SELECT query provided. Defaults to None (no table created).
  • decode_geom (bool,optional) – Decodes CARTO’s geometries into a Shapely object that can be used, for example, in GeoPandas. It only works for SELECT queries when is_select=True
  • is_select (bool,optional) – This argument has to be set depending on the query performed. True for SELECT queries, False for any other query. For the case of a SELECT SQL query ( is_select=True ) the result will be stored into a pandas DataFrame. When an arbitrary SQL query ( is_select=False ) it will perform a database operation (UPDATE, DROP, INSERT, etc.) By default is_select=None that means that the method will return a dataframe if the query starts with a select clause, otherwise it will just execute the query and return None

When is_select=True and the query is actually a SELECT query this method returns a pandas DataFrame representation of query supplied otherwise returns None. Pandas data types are inferred from PostgreSQL data types. In the case of PostgreSQL date types, dates are attempted to be converted, but on failure a data type ‘object’ is used.

CartoException – If there’s any error when executing the query

Query a table in CARTO and write a new table that is result of query. This query gets the 10 highest values from a table and returns a dataframe, as well as creating a new table called ‘top_ten’ in the CARTO account.

This query joins points to polygons based on intersection, and aggregates by summing the values of the points in each polygon. The query returns a dataframe, with a geometry column that contains polygons and also creates a new table called ‘points_aggregated_to_polygons’ in the CARTO account.

Updates the column my_column in the table my_table

Produce a CARTO map visualizing data layers.

Create a map with two data Layer s, and one BaseMap layer:

Create a snapshot of a map at a specific zoom and center:

List of zero or more of the following:

    : cartoframes Layer object for visualizing data from a CARTO table. See Layer for all styling options. : Basemap for contextualizng data layers. See BaseMap for all styling options. : Layer from an arbitrary query. See QueryLayer for all styling options.

Interactive maps are rendered as HTML in an iframe , while static maps are returned as matplotlib Axes objects or IPython Image.

IPython.display.HTML or matplotlib Axes

Find all boundaries available for the world or a region . If boundary is specified, get all available boundary polygons for the region specified (if any). This method is espeically useful for getting boundaries for a region and, with CartoContext.data and CartoContext.data_discovery , getting tables of geometries and the corresponding raw measures. For example, if you want to analyze how median income has changed in a region (see examples section for more).

Find all boundaries available for Australia. The columns geom_name gives us the name of the boundary and geom_id is what we need for the boundary argument.

Get the boundaries for Australian Postal Areas and map them.

Get census tracts around Idaho Falls, Idaho, USA, and add median income from the US census. Without limiting the metadata, we get median income measures for each census in the Data Observatory.

Region where boundary information or, if boundary is specified, boundary polygons are of interest. region can be one of the following:

  • table name (str): Name of a table in user’s CARTO account
  • bounding box (list of float): List of four values (two lng/lat pairs) in the following order: western longitude, southern latitude, eastern longitude, and northern latitude. For example, Switzerland fits in [5.9559111595,45.8179931641,10.4920501709,47.808380127]

If boundary is specified, then all available boundaries and accompanying geom_refs in region (or the world if region is None or not specified) are returned. If boundary is not specified, then a DataFrame of all available boundaries in region (or the world if region is None )

Discover Data Observatory measures. This method returns the full Data Observatory metadata model for each measure or measures that match the conditions from the inputs. The full metadata in each row uniquely defines a measure based on the timespan, geographic resolution, and normalization (if any). Read more about the metadata response in Data Observatory documentation.

Internally, this method finds all measures in region that match the conditions set in keywords , regex , time , and boundaries (if any of them are specified). Then, if boundaries is not specified, a geographical resolution for that measure will be chosen subject to the type of region specified:

  1. If region is a table name, then a geographical resolution that is roughly equal to region size / number of subunits .
  2. If region is a country name or bounding box, then a geographical resolution will be chosen roughly equal to region size / 500 .

Since different measures are in some geographic resolutions and not others, different geographical resolutions for different measures are oftentimes returned.

To remove the guesswork in how geographical resolutions are selected, specify one or more boundaries in boundaries . See the boundaries section for each region in the Data Observatory catalog.

The metadata returned from this method can then be used to create raw tables or for augmenting an existing table from these measures using CartoContext.data . For the full Data Observatory catalog, visit https://cartodb.github.io/bigmetadata/. When working with the metadata DataFrame returned from this method, be careful to only remove rows not columns as CartoContext.data <cartoframes.context.CartoContext.data> generally needs the full metadata.

Narrowing down a discovery query using the keywords , regex , and time filters is important for getting a manageable metadata set. Besides there being a large number of measures in the DO, a metadata response has acceptable combinations of measures with demonimators (normalization and density), and the same measure from other years.

For example, setting the region to be United States counties with no filter values set will result in many thousands of measures.

Get all European Union measures that mention freight .

Information about the region of interest. region can be one of three types:

  • region name (str): Name of region of interest. Acceptable values are limited to: ‘Australia’, ‘Brazil’, ‘Canada’, ‘European Union’, ‘France’, ‘Mexico’, ‘Spain’, ‘United Kingdom’, ‘United States’.
  • table name (str): Name of a table in user’s CARTO account with geometries. The region will be the bounding box of the table.

Note

If a table name is also a valid Data Observatory region name, the Data Observatory name will be chosen over the table.

Note

Geometry levels are generally chosen by subdividing the region into the next smallest administrative unit. To override this behavior, specify the boundaries flag. For example, set boundaries to 'us.census.tiger.census_tract' to choose US census tracts.

A dataframe of the complete metadata model for specific measures based on the search parameters.

  • ValueError – If region is a list and does not consist of four elements, or if region is not an acceptable region
  • CartoException – If region is not a table in user account

Get an augmented CARTO dataset with Data Observatory measures. Use CartoContext.data_discovery to search for available measures, or see the full Data Observatory catalog. Optionally persist the data as a new table.

Get a DataFrame with Data Observatory measures based on the geometries in a CARTO table.

Pass in cherry-picked measures from the Data Observatory catalog. The rest of the metadata will be filled in, but it’s important to specify the geographic level as this will not show up in the column name.

  • table_name (str) – Name of table on CARTO account that Data Observatory measures are to be added to.
  • metadata (pandas.DataFrame) – List of all measures to add to table_name . See CartoContext.data_discovery outputs for a full list of metadata columns.
  • persist_as (str,optional) – Output the results of augmenting table_name to persist_as as a persistent table on CARTO. Defaults to None , which will not create a table.
  • how (str,optional) – Not fully implemented. Column name for identifying the geometry from which to fetch the data. Defaults to the_geom , which results in measures that are spatially interpolated (e.g., a neighborhood boundary’s population will be calculated from underlying census tracts). Specifying a column that has the geometry identifier (for example, GEOID for US Census boundaries), results in measures directly from the Census for that GEOID but normalized how it is specified in the metadata.

A DataFrame representation of table_name which has new columns for each measure in metadata .


Latest Press Coverage

Map of Life has been covered by newspapers, magazines and blogs from around the world. Here are a few selected quotes from their articles, as well as links to access each article in its entirety.

Study Projects Troubles for 1700 Vertebrate Species

In 1983, around 40,000 Nile lechwes (Kobus megaceros) roamed South Sudan and eastern Ethiopia. By 2060, this endangered population of African antelope may be on the brink of extinction. The cause? Loss of wild habitat.

This antelope is just one of hundreds of species that may be imperiled in the next four to five decades, according to a recent NASA-funded study. Researchers from Yale University examined the habitats of 19,400 species to learn how they might be affected by human land-use and encroachment, such as urban development and deforestation. They found that habitats for nearly 1,700 bird, mammal, and amphibian species are expected to shrink about 6 to 10 percent per decade by 2070, greatly increasing the risk of extinction for these animals.

“We all want to see economic progress and development, and that necessarily implies further human-induced changes to landscapes,” said Walter Jetz, co-author of the study and professor of ecology at Yale. “But unless potential impacts of this land use on biodiversity are known and addressed in some form, the long-term consequences could lead to species forever lost for future generations.”

The maps on this page show the potential decrease of suitable habitats for two vulnerable species. The map above shows the habitat change for the Nile lechwes from 2015 (left) to 2070 (right). The antelope species could lose approximately 70 percent of its suitable habitat and become “critically endangered” by 2070.

The map below shows the habitat of Oreophryne monticola, a frog endemic to Indonesia. The frog is currently listed as “endangered” and is predicted to lose more than 50 percent of its habitat in Lombok and Bali by 2070.

“If a country is projected to see a lot of change of swamps or forest to agriculture, this a good predictor that some species in that area are in jeopardy,” said Jetz. “That doesn’t mean these species are necessarily going to go extinct, but they are going to be put under pressure.”

According to the study, amphibians will be the most affected by human land use, followed by birds and mammals. Geographically, species living in South America, Southeast Asia, Central and East Africa, and Mesoamerica are expected to experience the most habitat loss and the greatest increase of extinction risk.

To make these predictions, Jetz and co-author Ryan Powers created a model that allowed them to analyze 2015 habitat conditions of about 19,400 species under anticipated changes in land-use in these areas.

To first estimate the area of suitable habitats in 2015, the team used several remote sensing layers. Elevation data came from Shuttle Radar Topography Mission (SRTM) and the Advanced Spaceborne Thermal Emission and Reflection Radiometer (ASTER). Tree cover data came from the Global Forest Change data set, which uses Landsat data to document global tree cover gains and losses.

The researchers then ran a model combining this habitat suitability information with future land-cover projections from the Land Use Harmonization data set in order to estimate decadal changes from 2015 to 2070. They ran the numbers under four different socioeconomic scenarios that would bring variations in land use. (The maps on this page show the “middle-of-the-road” economic scenario and assume no land will be recovered once destroyed.) Even in the best cases, many species are predicted to experience habitat losses by 2070.

“Even though we might see certain losses into the future no matter what we do,” said Jetz, “we can adjust to have the greatest chance of preserving life.” The study could help future conservation efforts by local, national, and international organizations.

The research by Jetz and Powers feeds into an initiative called the Map of Life, a NASA-funded public web platform designed to integrate large amounts of biodiversity and environmental data from researchers and citizen scientists. The global database aims to support a worldwide monitoring of species distributions.

NASA Earth Observatory images by Lauren Dauphin, using data from Powers, Ryan, et al. (2019).

Due to humans, extinction risk for 1,700 animal species to increase by 2070

As humans continue to expand our use of land across the planet, we leave other species little ground to stand on. By 2070, increased human land-use is expected to put 1,700 species of amphibians, birds, and mammals at greater extinction risk by shrinking their natural habitats, according to a study by Yale ecologists published in Nature Climate Change.

To make this prediction, the ecologists combined information on the current geographic distributions of about 19,400 species worldwide with changes to the land cover projected under four different trajectories for the world scientists have agreed on as likely. These potential paths represent reasonable expectations about future developments in global society, demographics, and economics.

“Our findings link these plausible futures with their implications for biodiversity,” said Walter Jetz, co-author and professor of ecology and evolutionary biology and of forestry and environmental studies at Yale. “Our analyses allow us to track how political and economic decisions — through their associated changes to the global land cover — are expected to cause habitat range declines in species worldwide.”

"While biodiversity erosion in far-away parts of the planet may not seem to affect us directly, its consequences for human livelihood can reverberate globally." -Walter Jetz

The study shows that under a middle-of-the-road scenario of moderate changes in human land-use about 1,700 species will likely experience marked increases in their extinction risk over the next 50 years: They will lose roughly 30-50% of their present habitat ranges by 2070. These species of concern include 886 species of amphibians, 436 species of birds, and 376 species of mammals — all of which are predicted to have a high increase in their risk of extinction.

Among them are species whose fates will be particularly dire, such as the Lombok cross frog (Indonesia), the Nile lechwe (South Sudan), the pale-browed treehunter (Brazil) and the curve-billed reedhaunter (Argentina, Brazil, Uruguay) which are all predicted to lose around half of their present day geographic range in the next five decades. [These projections and all other analyzed species can be examined at the Map of Life website] (https://mol.org/species/projection/landuse).

“The integration of our analyses with the Map of Life can support anyone keen to assess how species may suffer under specific future land-use scenarios and help prevent or mitigate these effects,” said Ryan P. Powers, co-author and former postdoctoral fellow in the Jetz Lab at Yale.

Species living in Central and East Africa, Mesoamerica, South America, and Southeast Asia will suffer the greatest habitat loss and increased extinction risk. But Jetz cautioned the global public against assuming that the losses are only the problem of the countries within whose borders they occur.

“Losses in species populations can irreversibly hamper the functioning of ecosystems and human quality of life,” said Jetz. “While biodiversity erosion in far-away parts of the planet may not seem to affect us directly, its consequences for human livelihood can reverberate globally. It is also often the far-away demand that drives these losses — think tropical hardwoods, palm oil, or soybeans — thus making us all co-responsible.”

The study was funded by grants from the National Science Foundation, the Natural Sciences and Engineering Research Council of Canada, and the National Aeronautics and Space Administration.

Space-based tracker to give scientists a beyond-bird’s-eye-view of wildlife

The International Cooperation for Animal Research Using Space, or ICARUS, will be flying closer to the sun than ever when a pair of Russian cosmonauts installs the antennae for its state-of-the-art animal tracking system on the exterior of the International Space Station on Aug. 15. The installation will be one small step for the cosmonauts and one giant leap for Yale biodiversity research.

Thanks to the recently founded Max Planck-Yale Center (MPYC) for Biodiversity Movement and Global Change, Yale and U.S.-based biodiversity researchers will be among the first to make use of the big data that this groundbreaking scientific instrument will be collecting by early 2019.

For the past 16 years, ICARUS has been simultaneously developing the tiniest transmitters (by 2025, the team hopes to scale down solar-powered backpacks enough to fit them on desert locusts) and some of the most massive antennae (the equipment that the cosmonauts will be installing). Together, these two new technologies will give biodiversity researchers an unprecedented, extraterrestrial perspective on the lives of some of Earth’s smallest and most mobile creatures, such as fruit bats, baby turtles, parrots, and songbirds.

“The system represents a quantum leap for the study of animal movements and migration, and will enable real-time biodiversity monitoring at a global scale,” said Walter Jetz, professor of ecology and evolutionary biology at Yale and co-director of the MPYC.

"I expect ICARUS to exceed what has existed to date by at least an order of magnitude and someday potentially several orders." -walter jetz

“In the past, tracking studies have been limited to, at best, a few dozen simultaneously followed individuals, and the tags were large and readouts costly,” added Jetz. “In terms of scale and cost, I expect ICARUS to exceed what has existed to date by at least an order of magnitude and someday potentially several orders. This new tracking system has the potential to transform multiple fields of study.”

Even with the limited tracking technology available, biodiversity researchers have already been able to predict volcanic eruptions by tracking the movements of goat herds and understand impacts of climate change by following migration changes in birds. This new space station-based system will allow researchers to see “not only where an animal is but also what it is doing,” explained Martin Wikelski, chief strategist for ICARUS, director of the Max Planck Center for Ornithology, and co-director with Jetz of the MPYC.

“At a global scale, we will be able to monitor individual animal behaviors as well as get a grasp of their intricate life histories and interactions with each other,” said Wikelski. In addition to positional coordinates, the transmitters are able to capture each animal’s acceleration, alignment to the magnetic field of Earth, and moment-to-moment environmental conditions, including ambient temperature, air pressure, and humidity.

The technology provides an exciting tool to monitor changing wildlife and the connectivity of landscapes for conservation and public health, explained Jetz. Researchers will be able to apply this new language of mass animal movement to everything from greater forewarning of geological disasters, such as earthquakes and volcanic eruptions, to monitoring the next potential disease outbreak in humans. For example, Wikelski plans to use the new system to advance his own project of tracking the movement of African fruit bats as sentinels for finding the hosts of the Ebola virus. (Fruit bats have antibodies against, but do not transmit, this deadly disease.)

“Tracked animals can act as intelligent sensors and biological sentinels and in near real-time inform us about the biodiversity effects of ongoing environmental change,” explained Jetz.

By the beginning of 2019, Wikelski and colleagues will have 1,000 transmitters in the field, but eventually, they hope to grow that number to 100,000. Every time a transmitter enters the International Space Station’s beam — roughly four times daily — it may send up a data packet of 223 bytes. From there, the data will be relayed back to the ground station and subsequently distributed to research teams. All data — except sensitive conservation data such as rhino locations — will also be published on the publicly accessible database MoveBank, and will inform maps and trends in Map of Life, a web-based initiative headed by Jetz that integrates global biodiversity evidence.

As with all fields of scientific research, however, the data are only as good as their processing and analysis. MPYC will be the primary initial interpreter of the big data harvested by the ICARUS satellite. Fortunately, Jetz notes, with Yale’s investment in integrative data science as a research priority, MPYC can handle the big data sets that the ICARUS tracking system generates.

“Going back to my own Ph.D. work observing and tracking nocturnal birds in Africa with much inferior technology,” said Jetz, “I was always driven by the wish to document and understand biodiversity from the level of the individual up to the global scale.”

“The new technology will allow us to put the bigger picture together,” Jetz continued. “Thanks to the near-global scale of ICARUS and satellite-based remote sensing of the environment, we are finally able to connect individual behaviors and decisions with the use of space and environments at large scales. Our collaboration with Max Planck and ICARUS is a wonderful enabler of and complement to our work at Yale.”

Mapping Species for Half-Earth

The Significance of Biodiversity

Biodiversity is the variety of life on Earth, the building blocks of functioning ecosystems that provide the natural services on which all life depends, including people. Species, the fundamental units of biodiversity, are in the midst of an extinction crisis, losing ground globally at a rate 1,000 times greater than at any time in human history due to factors like habitat loss and climate change.

How do we stop this? Knowing where species live and the pressures threatening them is paramount in reversing the extinction crisis and maintaining the health of our planet, for ourselves and for future generations. As the impact of humans increasingly encroaches on critical habitats everywhere, determining ‘where’ to protect is just as critical as ‘how much’ to protect.

The Half-Earth Project In his book, Half-Earth, acclaimed biologist Edward O. Wilson proposed a solution commensurate with the problem: conserve half the Earth’s land and sea to protect the bulk of biodiversity from extinction. Scientists agreed that this proposal was both necessary and possible.

“In order to stave off the mass extinction of species, including our own, we must move swiftly to preserve the biodiversity of our planet.” — E.O. Wilson Born from his book and built on a solid scientific foundation, the Half-Earth Project is working to conserve half the Earth by protecting sufficient habitat to reverse the species extinction crisis and ensure the long-term health of our planet.

The next question the Half-Earth Project needed to answer was, which Half?

Enter Map of Life. Map of Life is a core tool of the Half-Earth Project, which is working to identify and prioritize areas of greatest biodiversity value, and communicate this information in new, dynamic, and engaging ways.

Based out of Yale University and the University of Florida, Map of Life assembles, integrates, and analyzes data on global species distributions. It brings together a wealth of information, assessing information on nearly 100,000 species from hundreds of data sources with multiple data types.

Building on several years of close collaboration with Google, Map of Life leverages Google Cloud Platform services to support biodiversity research, monitoring, education, and decision-making. By leveraging an enormous biodiversity database and a suite of spatial modeling tools, Map of Life is able to capture detailed patterns of species distributions at planetary scale.

Today, the Half-Earth Project Map is using new and existing data and applying cutting-edge capabilities of Google Cloud Platform with the goal of mapping terrestrial, marine, and freshwater species at up to 1 kilometer resolution.

Science-Driven Conservation: A big data problem using Google Cloud Platform solutions Data Warehousing For an initial set of analyses we leveraged the PostGIS suite of spatial functions to measure expected species presence by overlaying species range map polygons with a global grid composed of approximately 110 km x 110 km cells.

Outputs from these intersections are stored on Google Cloud Storage and imported to the BigQuery data warehousing service. The speed at which BigQuery can aggregate across large tables and compute metrics has been vital for analyzing large volumes of biodiversity data. This is increasingly important as the Half-Earth Project continues to add new species groups and generates higher resolution predictions of where species occur. Tables currently in the hundreds of millions of rows may scale to billions or trillions of rows as our taxonomic and spatial resolution increases. The low storage cost and transaction-based pricing BigQuery offers allows us to query and aggregate tables of such size without the maintenance and overhead required by a traditional data warehouse solution.

Measuring Biodiversity One aspect of our work measured biodiversity data in two ways: richness and range rarity.

Richness is the number of species occurring, or expected to occur, within a given area. Richness is the simplest way to measure biodiversity.

Range rarity is a continuous metric of range-restrictedness and crucial for considering species with very small ranges that are often of greatest conservation concern. Range rarity is a close proxy for the irreplaceability of a location when the goal is to conserve as many species as possible.

Determining Goals for Species Protection Another aspect of our analysis included estimating the amount of land already protected within any given ca. 100km cells grid cell (mentioned above). To map the protected area network, we filtered the World Database of Protected Areas, which became available as an Earth Engine public table asset in 2017, to remove redundant reserves and so called “paper-parks” that lack on-the-ground biodiversity protection. For areas that lack a geometry and only include a point location and reserve area, we generated a polygon by buffering the point provided to the size of the park. We then computed the area protected within each grid cell, and exported the results to Cloud Storage.

While no habitat loss is ideal, species with larger range sizes can generally afford to lose more habitat than those with smaller ranges. Accordingly, we determined individual species “protected” status based on range size and the proportion of their range that is protected. As the amount of protected area increases, the number of species protected also increases.

But what is good for one species group is not necessarily ideal for another, and what is good for the whole of biodiversity may leave small groups vulnerable.

To address this, we leveraged BigQuery’s support for user defined functions (UDF) and computed adequate protection levels with a simple javascript function. Given the ability to apply this test to each species, we ranked grid cells for each species group and tested to see how many species’ global protections met the criteria via a BigQuery window function run across the ranked grid.

Progress toward Half-Earth will be measured as a running total of conservation protections, with the ultimate goal being half the Earth’s land and sea. BigQuery window functions allow this to be computed quickly for each cell in the grid. As we selected breakpoints up to 50% of the Earth’s area, we tested each species with our function to see whether it met its minimum protected area, then counted the number of species that met the criteria for each step in the scenario.

Putting It Together To ensure rapid map tile delivery to the globe, we generated and exported static tilesets to Cloud Storage using Earth Engine’s Export.map.toCloudStorage() feature. This meant exporting compiled data as CSV from BigQuery, which can be re-joined to the grid Shapefile using OGR command line tools and ingested to Earth Engine as a table asset. From there we were able to visualize and explore the data in the Earth Engine code editor and export tiles once we were satisfied with the appearance.

Finally, with our Cloud Storage bucket populated with map tiles and data to drive charts and infographics, the front-end wizards at Vizzuality plugged in to our API to bring it all to life on the Half-Earth Map.

The Half-Earth Map pieces together species distribution data, the protected areas map, and a mask of human activities into a single map useful to scientists, conservationists, communities, decision-makers and anyone interested in biodiversity and the health of our planet.

Mapping Earth’s Species to Identify Conservation Priorities

THE HALF-EARTH PROJECT IS ‘UNLOCKING A NEW ERA IN DATA-DRIVEN CONSERVATION.’ The Half-Earth Project has launched online the first phase of their cutting-edge global biodiversity map. This unique, interactive asset uses the latest science and technology to map thousands of species around the world and illuminate where future conservation efforts should be located to best care for our planet and ourselves.

“The extinction of species by human activity continues to accelerate, fast enough to eliminate more than half of all species by the end of this century,” said E.O. Wilson in the New York Times Sunday Review on March 3. “We have to enlarge the area of Earth devoted to the natural world enough to save the variety of life within it. The formula widely agreed upon by conservation scientists is to keep half the land and half the sea of the planet as wild and protected from human intervention or activity as possible.”

Born from Wilson’s book Half-Earth: Our Planet’s Fight for Life, the Half-Earth Project is providing the urgently needed research, leadership and knowledge necessary to conserve half the planet’s surface. The new map sits at the center of this effort. By mapping the biodiversity of our planet, we can identify the best places to conserve to safeguard the maximum number of species.

“The Half-Earth Project is mapping the fine distribution of species across the globe to identify the places where we canprotect the highest number of species,” Wilson said. “By determining which blocks of land and sea we can string together for maximum effect, we have the opportunity to support the most biodiverse places in the world as well as the people who call these paradises home.”

The Half-Earth Project is targeting completion of the fine-scale species distribution map for most known terrestrial, marine, and freshwater plant and animal species within 5 years.

“This mapping tool is unlocking a new era in data-driven conservation,” said Paula Ehrlich, President and CEO of the E.O. Wilson Biodiversity Foundation and head of the Half-Earth Project. “It will provide the scientific foundation upon which communities, scientists, conservationists and decision-makers can achieve the goal of Half-Earth.”

A $5 million leadership gift from E.O. Wilson Biodiversity Foundation board member Jeff Ubben and his wife Laurie will seed the second phase of the Half-Earth Project’s mapping effort.

“Half-Earth can’t wait. We have to work quickly and we need to be smart about how we do the work,” said Jeff Ubben. “This map will give us the information we need to make strong conservation investments.”

“Ed Wilson framed Half-Earth as a moonshot necessary to preserve the future health of our planet,” Ehrlich said. “The investment in our work from Jeff and Laurie Ubben attaches rocket boosters to this moonshot.”

The 8 Million Species We Don't Know

The history of conservation is a story of many victories in a losing war. Having served on the boards of global conservation organizations for more than 30 years, I know very well the sweat, tears and even blood shed by those who dedicate their lives to saving species. Their efforts have led to major achievements, but they have been only partly successful.

The extinction of species by human activity continues to accelerate, fast enough to eliminate more than half of all species by the end of this century. Unless humanity is suicidal (which, granted, is a possibility), we will solve the problem of climate change. Yes, the problem is enormous, but we have both the knowledge and the resources to do this and require only the will.

The worldwide extinction of species and natural ecosystems, however, is not reversible. Once species are gone, they’re gone forever. Even if the climate is stabilized, the extinction of species will remove Earth’s foundational, billion-year-old environmental support system. A growing number of researchers, myself included, believe that the only way to reverse the extinction crisis is through a conservation moonshot: We have to enlarge the area of Earth devoted to the natural world enough to save the variety of life within it.

The formula widely agreed upon by conservation scientists is to keep half the land and half the sea of the planet as wild and protected from human intervention or activity as possible. This conservation goal did not come out of the blue. Its conception, called the Half-Earth Project, is an initiative led by a group of biodiversity and conservation experts (I serve as one of the project’s lead scientists). It builds on the theory of island biogeography, which I developed with the mathematician Robert MacArthur in the 1960s.

Island biogeography takes into account the size of an island and its distance from the nearest island or mainland ecosystem to predict the number of species living there the more isolated an ecosystem, the fewer species it supports. After much experimentation and a growing understanding of how this theory works, it is being applied to the planning of conservation areas.

So how do we know which places require protection under the definition of Half-Earth? In general, three overlapping criteria have been suggested by scientists. They are, first, areas judged best in number and rareness of species by experienced field biologists second, “hot spots,” localities known to support a large number of species of a specific favored group such as birds and trees and third, broad-brush areas delineated by geography and vegetation, called ecoregions.

All three approaches are valuable, but applying them in too much haste can lead to fatal error. They need an important underlying component to work — a more thorough record of all of Earth’s existing species. Making decisions about land protection without this fundamental knowledge would lead to irreversible mistakes.

The most striking fact about the living environment may be how little we know about it. Even the number of living species can be only roughly calculated. A widely accepted estimate by scientists puts the number at about 10 million. In contrast, those formally described, classified and given two-part Latinized names (Homo sapiens for humans, for example) number slightly more than two million. With only about 20 percent of its species known and 80 percent undiscovered, it is fair to call Earth a little-known planet.

Paleontologists estimate that before the global spread of humankind the average rate of species extinction was one species per million in each one- to 10-million-year interval. Human activity has driven up the average global rate of extinction to 100 to 1,000 times that baseline rate. What ensues is a tragedy upon a tragedy: Most species still alive will disappear without ever having been recorded. To minimize this catastrophe, we must focus on which areas on land and in the sea collectively harbor the most species.

Building on new technologies, and on the insight and expertise of organizations and individuals who have dedicated their lives the environment, the Half-Earth Project is mapping the fine distribution of species across the globe to identify the places where we can protect the highest number of species. By determining which blocks of land and sea we can string together for maximum effect, we have the opportunity to support the most biodiverse places in the world as well as the people who call these paradises home. With the biodiversity of our planet mapped carefully and soon, the bulk of Earth’s species, including humans, can be saved.

By necessity, global conservation areas will be chosen for what species they contain, but in a way that will be supported, and not just tolerated, by the people living within and around them. Property rights should not be abrogated. The cultures and economies of indigenous peoples, who are de facto the original conservationists, should be protected and supported. Community-based conservation areas and management systems such as the National Natural Landmarks Program, administered by the National Park Service, could serve as a model.

To effectively manage protected habitats, we must also learn more about all the species of our planet and their interactions within ecosystems. By accelerating the effort to discover, describe and conduct natural history studies for every one of the eight million species estimated to exist but still unknown to science, we can continue to add to and refine the Half-Earth Project map, providing effective guidance for conservation to achieve our goal.

The best-explored groups of organisms are the vertebrates (mammals, birds, reptiles, amphibians, fishes), along with plants, especially trees and shrubs. Being conspicuous, they are what we familiarly call “wildlife.” A great majority of other species, however, are by far also the most abundant. I like to call them “the little things that run the world.” They teem everywhere, in great number and variety in and on all plants, throughout the soil at our feet and in the air around us. They are the protists, fungi, insects, crustaceans, spiders, pauropods, centipedes, mites, nematodes and legions of others whose scientific names are seldom heard by the bulk of humanity. In the sea and along its shores swarm organisms of the other living world — marine diatoms, crustaceans, ascidians, sea hares, priapulids, coral, loriciferans and on through the still mostly unfilled encyclopedia of life.

Do not call these organisms “bugs” or “critters.” They too are wildlife. Let us learn their correct names and care about their safety. Their existence makes possible our own. We are wholly dependent on them.

With new information technology and rapid genome mapping now available to us, the discovery of Earth’s species can now be sped up exponentially. We can use satellite imagery, species distribution analysis and other novel tools to create a new understanding of what we must do to care for our planet. But there is another crucial aspect to this effort: It must be supported by more “boots on the ground,” a renaissance of species discovery and taxonomy led by field biologists.

Within one to three decades, candidate conservation areas can be selected with confidence by construction of biodiversity inventories that list all of the species within a given area. The expansion of this scientific activity will enable global conservation while adding immense amounts of knowledge in biology not achievable by any other means. By understanding our planet, we have the opportunity to save it.

As we focus on climate change, we must also act decisively to protect the living world while we still have time. It would be humanity’s ultimate achievement.

Map of Life: Preserving biodiversity with data

About Map of Life Map of Life supports global biodiversity education, monitoring, research, and decision-making by integrating and analyzing global information about species distributions and dynamics. Using hosted cloud technology, Map of Life makes its data available to scholars, researchers, students, teachers, and conservationists.

Industries: Education, Non-profit Location: United States Products: App Engine, BigQuery, Cloud SQL, Cloud Storage, Compute Engine, Google Earth Engine

Map of Life supports biodiversity education, monitoring, research, and decision-making by using Google Cloud Platform products to collect, analyze, and visually represent global data

Stores over 600M records for 44K+ species

Google Cloud Platform Results Speeds up data analysis required for accurate assessments of endangered species Provides scientific evidence to support conservation efforts Scales on demand to support additions and updates to large data volumes Stores over 600M records for 44K+ species

The richness and diversity of life on Earth is fundamental to the complex systems that inhabit it. But phenomena including climate change, pollution, unsustainable agriculture, and habitat destruction and degradation threaten the planet’s ecosystems and inhabitants. The World Wildlife Fund (WWF) estimates that wildlife populations worldwide declined by 58% between 1970 and 2012.

To reverse such trends, scientists, conservationists, and governments need to know where and how to target efforts to help prevent extinction and preserve biodiversity. Yale University and the University of Florida (UF) partnered to tackle the challenge by collecting and analyzing global sources of data and making information available to help guide research, policy, and conservation.

“Google Cloud Platform offers all the tools we need for large-scale data management and analysis. Its integration with Google Earth Engine makes it ideal for data visualization.” —Walter Jetz, Associate Professor, Department of Ecology and Evolutionary Biology, Yale University

Their solution, Map of Life, contains data about vertebrates, plants, and insects from international, national, and local sources, including BirdLife International, International Union for Conservation of Nature (IUCN), the Global Biodiversity Information Facility (GBIF). Yale and UF chose Google Cloud Platform to support Map of Life’s data storage, analysis, and mapping because of its superior ability to scale, integrate, manage, mine, and display data.

“Google Cloud Platform offers all the tools we need for large-scale data management and analysis,” says Walter Jetz, Associate Professor, Department of Ecology and Evolutionary Biology at Yale University. “Its integration with Google Earth Engine and Google Maps make it ideal for data visualization.”

Mapping where species are at-risk

Map of Life currently draws from more than 600 million records worldwide that contain information about approximately 44,351 species of vertebrates, plants, and insects and more than 700,000 species names, stores the information in Google Cloud Storage. Its scalable, high-performance architecture also includes the Google App Engine platform-as-a-service (PaaS) to host application logic and feed information to various user interfaces via RESTful application program interfaces (APIs). Geolocation and spatial data can be analyzed and displayed through Google App Engine APIs that connect to the Google Earth Engine, and CARTO geolocation data cloud platforms. The Map of Life CARTO service runs on Google Compute Engine virtual machines to improve scalability of on-demand vector mapping and query needs.

Combining data from multiple sources allows Map of Life to estimate the distribution of and trends within at-risk species and make this information available to naturalists, conservation groups, natural resource managers, scientists, and interested amateurs. Anyone who visits the website or downloads the mobile app can view the information.

Google Compute Engine performs complex data analyses to predict which species are at risk. The science behind understanding biodiversity and identifying and predicting trends within species requires multiple analytic iterations, each of which accounts for corrections and new input from scientists. Google Compute Engine is particularly well suited for this because of how quickly it processes each iteration.

“Map of Life uses Google BigQuery to analyze massive data sets, quickly. We can perform a query on 600 million species occurrence records in less than a minute, helping scientists reach conclusions more quickly.” —Jeremy Malczyk, Lead Software Engineer, Map of Life

“Every day we’re gathering more data, including from remote sensors,” says Walter. “Using Google Cloud Platform and Google Earth Engine, we’re able to make more accurate predictions about at-risk species worldwide.”

Map of Life continually incorporates new data sets, including those from individual observers who submit observations about vertebrates, plants, and insects. The platform even integrates remote-sensing data from Google Earth Engine and uses Google BigQuery to analyze large sets of unstructured data.

“Map of Life uses Google BigQuery to analyze massive data sets, quickly,” says Jeremy Malczyk, Lead Software Engineer for Map of Life. “We can perform a query on 600 million species occurrence records in less than a minute, helping scientists reach conclusions more quickly.”

Global conservation powered by data More than 100,000 scientists and concerned citizens already use Map of Life for biodiversity research and discovery. Map of Life is also developing new tools and visualizations to support the specific needs of government agencies and conservation organizations to help support the development of environmental policy.

The Chicago Field Museum and Map of Life received a $300,000 MacArthur Foundation grant to support conservation efforts in South America. Map of Life is creating data visualization dashboards for park managers, who can use the biodiversity information to improve conservation strategies in protected areas across Colombia, Ecuador, and Peru.

“Humankind has historically had very little information about biodiversity with sufficient spatial detail at a global scale. Map of Life is setting out to change that,” adds Walter. “By combining in data and models we aim to help everyone from eco-tourists who can appreciate biodiversity wherever they travel, to resource managers who need to handle development in a sustainable way, and to governments who want to protect biodiversity.”

Map of Life: new global biodiversity patterns pages launched

Map of Life has released a first suite of maps that aggregate biodiversity patterns. These pages feature global biodiversity patterns based on publications in Nature and PNAS. The types of maps available are:

Species Phylogenetic Functional
Species Richness Phylogenetic Diversity Functional Diversity
Species Endemism Phylogenetic Endemism Functional Endemism
Local Species Diversity Priority Local Phylogenetic Diversity Priority Local Functional Diversity Priority
Global Species Diversity Priority Global Phylogenetic Diversity Priority Global Functional Diversity Priority

Explore these new resources here.

Study: Targeted conservation could protect more of Earth's biodiversity

Research by Yale and the University of Grenoble documents where additional conservation efforts globally could most effectively support the safeguarding of species (some examples shown here) that are particularly distinct in their functions or their position in the family tree of life. (Image credits: family tree illustration, Laura Pollock Solenodon paradoxus photo, Nate Upham additional photos, Wikipedia.) A new study finds that major gains in global biodiversity can be achieved if an additional 5% of land is set aside to protect key species.

Scientists from Yale University and the University of Grenoble said such an effort could triple the protected range of those species and safeguard their functional diversity. The findings underscore the need to look beyond species numbers when developing conservation strategies, the researchers said.

“Biodiversity conservation has mostly focused on species, but some species may offer much more critical or unique functions or evolutionary heritage than others — something current conservation planning does not readily address,” said Walter Jetz, a Yale associate professor of ecology and evolutionary biology, and director of the Yale Center for Biodiversity and Global Change.

“We show that a direct consideration of these other biodiversity facets identifies different regions as high-priority for conservation than a focus on species does, and more effectively safeguards functions or evolutionary heritage,” Jetz said. “We find that through the smart placement of conservation areas, strong gains in the conservation of the multiple facets of biodiversity facets are possible.”

The study appears online May 24 in the journal Nature. Laura Pollock of the University of Grenoble is the study’s first author, Jetz is senior author, and Wilfried Thuiller of the University of Grenoble is co-author.

The researchers noted that 26% of the world’s bird and mammal species are not reliably included in protected areas. The outlook for filling gaps in bird and mammal diversity could improve dramatically by smartly expanding the areas currently managed for conservation, they said.

The researchers advocate a conservation strategy that emphasizes global representation, i.e., the planetary safeguarding of species function or evolutionary heritage planet-wide, rather than local representation. They estimate that a carefully prepared 5% increase in conservation area would allow a dramatically improved capture of bird and mammal biodiversity facets an approach focused on species numbers alone would be much less optimal, the researchers said.

Jetz and his colleagues also said their approach enables a more comprehensive guidance and capture of progress as mandated by the Convention on Biological Diversity and under evaluation by the Intergovernmental Science-Policy Platform on Biodiversity and Ecosystem Services.

“Given the current biodiversity crisis, these results are encouraging because they show big conservation gains are possible for aspects of biodiversity that might otherwise be overlooked in conservation plans,” Pollock said. “This biodiversity is key to retaining the tree of life or functioning ecosystems, which nicely fits declared international policy goals. This approach can be updated and refined as the world’s biodiversity becomes better understood, catalogued, and documented.”

The researchers have created interactive web maps in the Map of Life project to accompany the study. They can be found at: https://mol.org/patterns/facets

The National Science Foundation, the Yale Center for Biodiversity and Global Change, the People’s Programme of the European Union’s Seventh Framework Programme, and the European Research Council supported the research.

How Map of Life helps preserve Earth’s biodiversity with Google Cloud Platform

Global biodiversity is declining at an unprecedented rate: The World Wildlife Foundation estimates the decline of two thirds of the earth's vertebrate populations by 2020, providing supporting evidence that we are currently in an extinction crisis. With so many species at risk, it’s difficult for scientists, conservationists and government agencies to know how and where to prioritize and target efforts to halt extinctions and preserve biodiversity.

Map of Life, a collaborative project hosted by Yale University and the University of Florida, endeavors to tackle this challenge by providing comprehensive biodiversity information that integrates data from a large number of sources, including museums, conservation groups, government agencies and individuals. It layers an astounding amount of information for hundreds of thousands of species and using a rapidly growing amount of data, currently over 600 million records, on web and mobile-based Google Maps.

Map of Life needed a cloud-based platform to host their considerably large database, and chose Google Cloud Platform because it offers the best scalable tools to integrate, manage, mine and display data, additionally integrating products with Google Earth Engine and Google Maps.

“Google Cloud Platform offers all the tools we needed for large-scale data management and analysis. Its integration with Google Earth Engine and Google Maps make it ideal for data visualization.”

— Walter Jetz, Associate Professor, Department of Ecology and Evolutionary Biology, Yale University.

Pinpointing at-risk species

Combining data from multiple sources, Map of Life provides estimates of the distribution and potential trends of at-risk species and makes this information available to naturalists, conservation groups, resource managers, and global threat assessors. Anyone who visits the website or downloads the app can get information about which species occur where, globally. Google Cloud Storage hosts the data and scales automatically as the amount of information grows. Google App Engine runs a middleware API that integrates the data, makes it available to researchers, and displays it on Google Maps. Google Compute Engine performs data analyses to predict which species are at risk.

In biodiversity science, making predictions is extremely complex and requires many iterations, each of which includes corrections and new input from scientists. Compute Engine is particularly well-suited for this because of how quickly it performs each iteration. That allows more iterations to be run in a given time period.

“Every day we’re gathering more data, including from remote sensors. Using Google Cloud Platform and Google Earth Engine, we’re able to make more accurate predictions about at-risk species worldwide,” says Jetz.

Map of Life continually incorporates new data sets from resources around the world. It also allows people to send in their own observations about birds, mammals, cacti, and many more organisms. This platform integrates remote-sensing data from Google Earth Engine additionally utilizing Google BigQuery to perform queries on very large data sets.

“Map of Life uses Google BigQuery to analyse massive data sets, quickly. We can perform a query on 600 million species occurrence records in less than a minute, helping scientists reach conclusions more quickly,” says Jeremy Malczyk, lead software engineer for Map of Life.

Addressing biodiversity and conservation across the globe

Well over 100,000 scientists and citizen scientists are using Map of Life for biodiversity research and discovery. Map of Life is now developing visualizations and tools to support the specific needs of government agencies and conservation bodies and to help decision-making for environmental policy. In September 2016, the Chicago Field Museum and Map of Life received a $300,000 MacArthur Foundation grant to support conservation efforts in South America. Dashboards will provide park managers with biodiversity information including lists of species expected in a particular location. That information will be used to improve conservation strategies in protected areas within South America.

“Humankind has historically had very little information about biodiversity with sufficient spatial detail at a global scale. With Map of Life are setting out to change that. By combining in data and models we aim to help everyone from ecotourists who can appreciate biodiversity wherever they travel, to resource managers who need to handle development in a sustainable way, and to governments who want to protect biodiversity,” Jetz says.

Global Mountain Biodiversity Assessment Mountain Portal: a powerful new online tool developed by Map of Life for exploring mountain biodiversity

The Global Mountain Biodiversity Assessment (GMBA) teamed up with Map of Life (MOL) to launch a new web-portal for the visualization and exploration of biodiversity data for over 1000 mountain ranges defined worldwide.

Mountains are hotspots of biodiversity and areas of high endemism that support one third of terrestrial species and numerous ecosystem services. Mountain ecosystems are therefore of prime importance not only for biodiversity, but for human well-being in general. Because of their geodiversity, mountain ecosystems have served as refuge for organisms during past climatic changes and are predicted to fulfill this role also under forthcoming changes. Yet, mountains are responding to increasing land use pressure and changes in climatic conditions, and collecting, consolidating, and standardizing biodiversity data in mountain regions is therefore important for improving our current understanding of biodiversity patterns and predicting future trends.

In order to accurately predict potential changes in mountain biodiversity in response to drivers of global changes and develop sustainable management and conservation strategies, we must be able to define what exactly a mountain is, where mountains are in the world, and what species currently occur in those mountains.

More than 1000 mountain ranges around the world have now been described in a new study published in Alpine Botany by Christian Körner et al. (2016). Additionally, and for the first time, this global mountain inventory coverage has also been combined with expert range maps for approximately 60,000 species across different organismic groups and is being made available online through the Mountain Portal. The Mountain Portal is an interactive web platform provided by the Global Mountain Biodiversity Assessment of Future Earth and developed by Map of Life. With just a few clicks users can explore and download growing lists of mountain ranges and expected species. Downloaded data can then be used for a multitude of projects ranging from mechanistic studies on the evolution and ecological drivers of mountain biodiversity to the development of indicators in sustainability research.

The mountain portal is an open source tool for all types of users, ranging from laymen and citizen scientists to researchers, practitioners, stakeholders and policy makers. It is an evolving resource that will utilize the power of the global community to improve mountain biodiversity and inventory information.

Map of Life and The Field Museum win MacArthur Foundation grant

The Yale-led Map of Life project and The Field Museum in Chicago have won a $300,000 MacArthur Foundation grant to support conservation decisions in South America.

The two institutions will work with park services and wildlife agencies in Peru, Ecuador, and Colombia to improve conservation actions by harnessing better information about biodiversity. They will develop online dashboards to provide park managers with demand-driven, actionable biodiversity information such as lists of species expected in a particular location and estimates of their distribution trends. Park staff and visitors will use the service, which will be tailored to each park reserve area.

Map of Life is an online platform for species distributions, led by Walter Jetz, associate professor of ecology and evolutionary biology, and director of the Yale Center for Biodiversity and Global Change. “The new tools will provide biodiversity information specific to single parks and offer reserve managers basic analysis and reporting tools on data gaps and actual conservation gaps in their reserve system,” Jetz said.

Map of Life’s partner in the project, The Field Museum, has helped protect 23 million acres of wilderness in the tropical Andes through its Keller Science Action Center.

For more information, visit:

Map of Life named a 2016 Best App for Teaching & Learning

The American Association of School Librarians has named Map of Life, a Yale-led project with the University of Florida that assembles and integrates multiple sources of data about species distributions worldwide, as a 2016 Best App for Teaching & Learning.

The recognition honors apps of “exceptional value to inquiry-based teaching and learning” by fostering innovation, creativity, active participation, and collaboration. The award was announced June 25 at the American Library Association annual conference in Orlando, Florida.

Walter Jetz, associate professor of ecology and evolutionary biology, and director of the Yale Program in Spatial Biodiversity, is the guiding force behind Map of Life.

The Map of Life app puts a wealth of the world’s knowledge about biodiversity in the palm of a user’s hand. It can tell users which species are likely to be found in the vicinity, with photos and text to help identify and learn about species. The app also helps users create personal lists of observations and contribute those lists to scientific research and conservation efforts.

Range geometry and socio-economics dominate species-level biases in occurrence information

Despite the central role of species distributions in ecology and conservation, occurrence information remains geographically and taxonomically incomplete and biased. Efforts to address this problem, such as targeted data mobilization and advanced distribution modelling, all crucially rely on a solid understanding of the patterns and determinants of occurrence information. Numerous socio-economic and ecological drivers of uneven record collection and mobilization among species have been suggested, but the generality of their effects remains untested. Here, we provide the first global analysis of patterns and drivers of species-level variation in different metrics of occurrence information.

A global cloud atlas for predicting biodiversity and ecoystems

A study by former postdoc Adam Wilson and Walter Jetz provides a new 1km map of global cloud cover variation that provides striking insights into the fine-scale variation of habitats and species. Published in PLoS Biology. To browse the map, see http://www.earthenv.org/cloud

A Cloud Atlas Provides Clues to Life on Earth

After countless years of daydreamers being told otherwise, there’s now a good reason to keep your head in the clouds. Scientists combed through satellite photographs of cloud cover taken twice a day for 15 years from nearly every square kilometer of Earth to study the planet’s varied environments.

By creating cloud atlases, the researchers were able to better predict the location of plants and animals on land with unprecedented spatial resolution, allowing them to study certain species, including those that are often in remote places. The results were published last week in PLOS Biology.

Clouds directly affect local climates, causing differences in soil moisture and available sunlight that drive photosynthesis and ecosystem productivity.

The researchers demonstrated the potential for modeling species distribution by studying the Montane woodcreeper, a South American bird, and the King Protea, a South African shrub.

“In thinking about conserving biodiversity, one of the most important scientific questions is ‘Where are the species?’” said Adam Wilson, an ecologist now at the University at Buffalo, who led the study. The maps also could help monitor ecosystem changes.

For cloud-gazing, you can download the data: earthenv.org/cloud.html

50 Säugetiere und 222 Vogelarten im Umkreis

Ein digitales Bestimmungsbuch, das über Tiere und Pflanzen Auskunft gibt - wo auch immer man sich aufhält: Die App "Map of Life" zeigt an, was rund um den eigenen Standort kreucht und fleucht. Die Nutzer können sogar die Wissenschaft voran bringen.

Map of Life, the biodiversity in your hand

Map of Life builds on a global scientific effort to help you discover, identify and record species worldwide.

Map of Life is an application about biodiversity that gathers data from observations and references from numerous databases throughout the world, with information on different animal groups and plants. Some differences were detected between the mobile and the web app version, so they will be described separately.

The mobile app offers a lateral menu with different options:

What’s around me, option based on the geolocation of the user, delimiting an area (the radius is unknown) on which the information is extracted. Search the map by entering a place name or clicking on the map. Search for species by common or scientific name. My records, where you can upload your sightings after opening a free account. Settings, section where you can select the language or access to the help (only in English), for example. The information for each species includes a data sheet with a description taken from Wikipedia (in the selected language for the app), a map of geographical distribution and multiple images, in addition to the classification of the IUCN Red List.

The web app is more complete than the mobile version, since from the “Detailed Map” option you can filter information such as the range of years or the uncertainty, related in this case with geolocation errors (less accurate in old sightings and more detailed in actual observations made with modern devices).

Map of Life based indicators supporting GEO BON, CBD

Last week, an ad-hoc technical expert group of the CBD met in Geneve to advise CBD on a small set of indicators that could be used to assess progress towards the Aichi targets. GEO BON presented a new generation of indicators based on integrating information from a small set of essential biodiversity variables (http://www.geobon.org/Downloads/brochures/2015/GBCI_Version1.1_low.pdf).

The indicators, developed in collaboration with GEO BON partners Map of Life and CSIRO, were the Species Habitat Indices (Target 5 and 12), the Biodiversity Habitat Index (Target 5), the Species Protection Index Target 11), the Protected Area Representativeness and Connectedness Indices (Target 11), the Global Ecosytem Restoration Index (Target 15), and the Species Status Information Index (Target 19). They are based on global datasets for 4 EBVs: Species Distributions, Taxonomic Diversity (gamma diversity), Ecosystem Extent, and Primary Productivity. The indicators were very well received at the AHTEG, and were adopted as specific examples of indicators for these targets. They also illustrate the power of EBVs as a modelled layer between direct observations and indicators and its potential to generate global indicators and spatial explicit datasets.

Research in the news: Spotting the knowledge gaps in biological species data

Wealthy, emerging countries that are home to some of the most threatened animals on Earth are also the very places where biological records about animals are most sparse.

A comprehensive survey of global species distribution data, conducted in conjunction with the Yale-based Map of Life team, found that few records are accessible for Brazil, China, India, and several other large, emerging economies.

The survey results appear in the journal Nature Communications. The scientists investigated millions of records about the distribution of all known species of mammals, birds, and amphibians. Most of the records came from natural history museums or regional field surveys that make such information available.

“In our study we show that in most parts of the world, and even in many well-off countries, way too little data has been collected or shared to guide conservation and sustainable resource use,” said co-author Walter Jetz, a Yale University associate professor of ecology and evolutionary biology, and director of the Yale Program in Spatial Biodiversity Science & Conservation.

Jetz is the guiding force behind Map of Life, a project that assembles and integrates multiple sources of data about species distributions worldwide, including a mobile app. “More international data sharing is critical, and efforts such as Map of Life can help guide the efficient collection and use of new information,” Jetz said.

The new study sheds light on exactly where species information is most needed. “Until now it was thought that the largest data gaps were in tropical developing countries, that are rich in biodiversity but often lack the resources to study it,” said lead author Carsten Meyer of the University of Göttingen. “Our study adjusts and refines this impression and demonstrates the need to carefully assess and close these gaps.”

Additional co-authors of the study are Holger Kreft of the University of Göttingen and Rob Guralnick of the University of Florida-Gainesville.

For more information about the Map of Life, visit the website.

Toute la vie dans une app

Avec Map of Life, le big data et la géocalisation s’allient pour faire de vous un explorateur averti, voire un biologiste enrichissant la plateforme de ses observations

Vous êtes-vous jamais demandé quels étaient tel étrange oiseau croisé en voyage, telle fleur rencontrée en randonnée ou tel batracien baillant au bord de l’étang? Grâce au big data et à la géolocalisation, il suffit désormais d’un smartphone pour se muer en explorateur averti du grand jardin de la vie. L’initiative en revient à Map of Life, un effort de recherche international coordonné par l’Université de Yale.

Une fois l’app téléchargée, il suffit d’activer la fonction «What’s around me» pour connaître la composition de la flore et la faune alentour. Sur l’île de Miyajima au Japon (où cette chronique a été écrite), on trouve par exemple 229 oiseaux, 39 mammifères, 3 tortues, 19 amphibiens, 48 coléoptères et 19 conifères. A chaque espèce correspondent une fiche signalétique, des images et une carte des zones d’habitat.

Il y a plus: Map of Life est mise à jour en fonction des dernières observations par satellite, études académiques, bases de données et recensements. L’utilisateur lui-même peut identifier des espèces et/ou découvrir leur présence en sauvegardant et partageant ses observations sur la plateforme. «Les changements environnementaux, les disparitions d’espèces ou les invasions sont des irrégularités qui restent difficiles à détecter par satellite», explique dans un article le professeur Walter Jetz du Département d'écologie et l'évolution de Yale. «Vos marches en forêt, dans le désert ou dans les prés pourraient devenir des ressources inestimables pour la compréhension locale et globale de la biodiversité.»

Map of Life, pour iPhone et Android. Gratuit

MOL app reviewed by Science Magazine

The field guide, rebooted. Map of Life joins a small but growing number of mobile applications seeking to reimagine the field guide by combining big data and mobile technology. Reviewer Gregory R. Goldsmith takes the app for a test drive, breaking down its pros, cons, and its potential for attracting a new generation of ecologists.

Map of Life: A phone app that helps track wildlife

Walter Jetz has been working on the Map of Life app for four years. He's an expert in biodiversity, teaching at both Yale University in the US and Imperial College in the UK. I caught up with him on a slightly overcast day on a picnic table in the leafy grounds of Silwood Park, Imperial College’s campus about 40km west of London, where he explained how the app works. Touching the light blue icon with a three-branched tree opens the Map of Life, clicking on "What's Around Me", brings up a list including amphibians, butterflies, bees and 180 birds. "We can move on to the songbirds, there are all sorts of tits and warblers around me that we can hear and then connect up with the app," Jetz told Al Jazeera. More than 20,000 people from Brazil to Indonesia to South Africa have downloaded the app since it launched two months ago. Jetz said five to 10,000 people log on daily. "The exciting thing about the app is it’s not just a field guide it’s a flipped field guide. So instead of you having to sift through pages and pages of a field guide to identify that species you’ve just seen," Jetz said. "It’s already a tailored list of species where you are right now." The app is an international collaboration with scientists and computer programmers funded in part by NASA and the National Science Foundation in the US. Valuable information There are 35,000 species on the app across the globe but it’s not just meant to be fun in the park science and biodiversity are the core. Users in remote parts of the world, particularly near the Equator, are encouraged to report back their sightings. Jetz said the citizen scientists are providing valuable information. He said app users in Java had spotted a turtle not seen for many years. "You’d be surprised for one or two of those (turtle) species we barely have any information at all," Jetz said. "We have a rough map but no points on the ground that would tell us in detail where the species would be found." The sightings are pinpointed by the phone's built-in GPS and added to the ever growing database. Jetz said the app comes at a critical time as many species are changing fast - some becoming dominant in a region, others moving to different areas. "Suddenly, we have information about potential threats, potential risks of extinction….which allows us to paint a detailed picture of where the species is and how its range may fair in the future," said Jetz, whose enthusiasm for wildlife was sparked by a childhood in Bavaria and is undimmed. The data can be passed on to policy makers who can then decide if another agriculture project in a region is a good idea or if it will harm nearby species and impact biodiversity. There are plans to add to the app’s six languages and within a few months there will be sounds the animals make to improve identification. The pictures and text are easy to understand I used the app in New York’s Central Park last month when I spotted a bright orangey-red bird with a matching beak, sure enough the Map of Life identified it as a Northern Cardinal.

Life? There’s an app for that.

When the ornithologist and painter Roger Tory Peterson published the first modern field guide in 1934, he solved one of the biggest problems in species identification. No longer was it necessary to shoot or catch a creature and take it to an expert to find out its name all you needed was the Field Guide to Birds (or one its many successors, which cover all sorts of living creatures)—along with sharp eyes, patience, and fast fingers to turn the pages.

Now, thanks to smartphones and “big data,” an effort is under way to create field guides that can tell you what’s around even before you lift the binoculars or the magnifying lens. A field guide of this sort was recently launched by Map of Life, an international research effort headquartered at Yale’s Program in Spatial Biodiversity Science and Conservation.

The Map of Life app shows what kinds of life surround you, no matter where you are in the world. When you request information on a particular group (mammals, birds, amphibians, trees, wildflowers, turtles, dragonflies, butterflies, or fish—a collection that is growing rapidly), the app delivers an inventory of all the species believed to be in your area. It also provides images, range maps, and authoritative natural history information, so you can satisfy your curiosity about the frog or butterfly you just glimpsed. The data are culled from scientific literature and public databases, along with satellite remote-sensing, so the app is constantly updated. The predictions of where and when species may occur are generated using the latest modeling techniques. The app also makes the study of natural history more interactive. Therein lies great scientific utility. The gaps and uncertainties in our data on spatial biodiversity can be huge, and they put constraints on scientific undertakings such as assessments of species status and trends, monitoring of species invasions, resource management, and ecological research. Changing environments and species losses and invasions are on the horizon, yet there is no satellite system or other system to monitor these perturbations and few means for biologists to assess and predict the consequences.

But detailed observations from individuals, especially in understudied regions, could advance our knowledge significantly. Armed with mobile technology, amateurs can become citizen scientists, sharing their observations and helping to fill in the gaps. Your hike into the woods, the desert, or the prairie could become an invaluable resource for our global and local understanding of biodiversity.


Another Word For It Patrick Durusau on Topic Maps and Semantic Diversity

GRASS GIS is very relevant for anyone wanting to use data science to plan protests.

You can plan a protest using corner store maps, but those are unlikely to have alleys, bus stops, elevation, litter cans, utilities, and other details.

Other participants will have all that data and more so evening up the odds is a good idea.

Apologizes for the long quote but I don’t know which features/capabilities of GRASS GIS will be most immediately relevant for you.

General Information

Geographic Resources Analysis Support System, commonly referred to as GRASS GIS, is a Geographic Information System (GIS) used for data management, image processing, graphics production, spatial modelling, and visualization of many types of data. It is Free (Libre) Software/Open Source released under GNU General Public License (GPL) >= V2. GRASS GIS is an official project of the Open Source Geospatial Foundation.

Originally developed by the U.S. Army Construction Engineering Research Laboratories (USA-CERL, 1982-1995, see history of GRASS 1.0-4.2 and 5beta), a branch of the US Army Corp of Engineers, as a tool for land management and environmental planning by the military, GRASS GIS has evolved into a powerful utility with a wide range of applications in many different areas of applications and scientific research. GRASS is currently used in academic and commercial settings around the world, as well as many governmental agencies including NASA, NOAA, USDA, DLR, CSIRO, the National Park Service, the U.S. Census Bureau, USGS, and many environmental consulting companies.

The GRASS Development Team has grown into a multi-national team consisting of developers at numerous locations.

In September 2006, the GRASS Project Steering Commitee was formed which is responsible for the overall management of the project. The PSC is especially responsible for granting SVN write access.

General GRASS GIS Features

GRASS GIS contains over 350 modules to render maps and images on monitor and paper manipulate raster, and vector data including vector networks process multispectral image data and create, manage, and store spatial data. GRASS GIS offers both an intuitive graphical user interface as well as command line syntax for ease of operations. GRASS GIS can interface with printers, plotters, digitizers, and databases to develop new data as well as manage existing data.

GRASS GIS and support for teams

GRASS GIS supports workgroups through its LOCATION/MAPSET concept which can be set up to share data and the GRASS installation itself over NFS (Network File System) or CIFS. Keeping LOCATIONs with their underlying MAPSETs on a central server, a team can simultaneously work in the same project database.


  1. Download data and transform data
    • Excel
  2. Find and download shapefiles
    • Census TIGER
  3. Import maps and join with data and style
    • ArcGIS or QGIS
  4. Export and tweak for style further
    • Tableau, CartoDB, Illustrator


Data Clients¶

SQLClient class is a client to run SQL queries in a CARTO account. It also provides basic SQL utilities for analyzing and managing tables.

credentials ( Credentials ) – A Credentials instance can be used in place of a username`|`base_url / api_key combination.

Run a SQL query. It returns a list with content of the response. If the verbose param is True it returns the full SQL response in a dict . For more information check the SQL API documentation <https://carto.com/developers/sql-api/reference/#tag/Single-SQL-Statement> .

Run a long running query. It returns an object with the status and information of the job. For more information check the Batch API documentation <https://carto.com/developers/sql-api/reference/#tag/Batch-Queries> .

query (str) – SQL query.

Get the distict values and their count in a table for a specific column.

Get the number of elements of a table.

table_name (str) – name of the table.

Get the bounds of the geometries in a table.

table_name (str) – name of the table containing a “the_geom” column.

Show information about the schema of a table.

Show information about a column in a specific table. It returns the COUNT of the table. If the column type is number it also returns the AVG, MIN and MAX.