Dynamic URL Store

This module contains the DynamicURLStore store that communicates with a remote HTTP server which provides the actual data storage. This is a store which implements the basic operations via HTTP GET, POST, PUT and DELETE commands as described in the class documentation.

The implementation relies on the third-party requests library to handle the HTTP operations.

class encore.storage.dynamic_url_store.DynamicURLStore(base_url, query_url, url_format='{base}/{key}/{part}', url_format_no_part='{base}/{key}', parts={'permissions': 'auth', 'data': 'data', 'metadata': 'metadata'})

Store implementation which gets and sets from a web server

This store expects a server which exposes URLs for each key. By default these URLs are of the form:

<base>/<key>/<part>

Where <base> is a common prefix, <key> is the key of interest, and <part> is one of “data”, “metadata” or “auth”. If the store does not follow this format, you can provide a differnt url_format argument and a different mapping of <part> to aspects of the key.

The server is expected to respond to queries against these URLS in the following ways:

GET <base>/<key>/data
return the bytes in the body of the response
PUT <base>/<key>/data
accept the data bytes from the body of the request
GET <base>/<key>/metadata
return metadata as JSON
PUT <base>/<key>/metadata
set the metadata based on JSON contained in the body of the request
POST <base>/<key>/metadata
update the metadata based on JSON contained in the body of the request (as dict.update())
GET <base>/<key>/auth
return permissions information as JSON
PUT <base>/<key>/auth
set the permissions based on JSON contained in the body of the request
POST <base>/<key>/metadata
update the permissions based on JSON contained in the body of the request

In addition, a DELETE request to a URL of the form <base>/<key> should remove the key from the remote store. This pattern is configurable via the url_format_no_part argument to the constructor.

In addition, the server should have a query URL which accepts GET reuqests containing a JSON data structure of metadata key, value pairs to filter with, and should return a list of macthing keys, one per line.

connect(credentials=None)

Connect to a DynamicURLStore

Parameters:credentials ((user_tag, requests.Session)) – The credentials are a tuple containing ther user’s permission tag and a requests Session initialized with appropriate authentication.
delete(key)

Delete a key from the repsository.

This may be left unimplemented by subclasses that represent a read-only key-value store.

Parameters:key (string) – The key for the resource in the key-value store. They key is a unique identifier for the resource within the key-value store.
Events:StoreDeleteEvent - On successful completion of a transaction, a StoreDeleteEvent should be emitted with the key.
get(key)

Retrieve a stream of data and metdata from a given key in the key-value store.

Parameters:key (string) – The key for the resource in the key-value store. They key is a unique identifier for the resource within the key-value store.
Returns:value (instance of Value) - An instance of a Value subclass which holds references to the data, metadata and other information about the key.
Raises:KeyError - If the key is not found in the store, a KeyError is raised.
get_data(key)

Retrieve a stream from a given key in the key-value store.

Parameters:key (string) – The key for the resource in the key-value store. They key is a unique identifier for the resource within the key-value store.
Returns:data (file-like) - A readable file-like object the that provides stream of data from the key-value store.
Raises:KeyError - This will raise a key error if the key is not present in the store.
get_metadata(key, select=None)

Retrieve the metadata for a given key in the key-value store.

Parameters:
  • key (string) – The key for the resource in the key-value store. They key is a unique identifier for the resource within the key-value store.
  • select (iterable of strings or None) – Which metadata keys to populate in the result. If unspecified, then return the entire metadata dictionary.
Returns:

metadata (dict) - A dictionary of metadata associated with the key. The dictionary has keys as specified by the select argument. If a key specified in select is not present in the metadata, then it will not be present in the returned value.
Raises:

KeyError - This will raise a key error if the key is not present in the store.
get_permissions(key)

Return the set of permissions the user has

Parameters:

key (str) – The key for the resource which you want to know the permissions.
Returns:

permissions (dict of str: set of str) - A dictionary whose keys are the permissions and values are sets of tags which have that permission.
Raises:
  • KeyError - This error will be raised if the key does not exist or the user is not authorized to see it.
  • AuthorizationError - This error will be raised if user is authorized to see the key, but is not an owner.
query(select=None, **kwargs)

Query for keys and metadata matching metadata provided as keyword arguments

This provides a very simple querying interface that returns precise matches with the metadata. If no arguments are supplied, the query will return the complete set of metadata for the key-value store.

Parameters:
  • select (iterable of strings or None) – An optional list of metadata keys to return. If this is not None, then the metadata dictionaries will only have values for the specified keys populated.
  • kwargs – Arguments where the keywords are metadata keys, and values are possible values for that metadata item.
Returns:

result (iterable) - An iterable of (key, metadata) tuples where metadata matches all the specified values for the specified metadata keywords. If a key specified in select is not present in the metadata of a particular key, then it will not be present in the returned value.
query_keys(**kwargs)

Query for keys matching metadata provided as keyword arguments

This provides a very simple querying interface that returns precise matches with the metadata. If no arguments are supplied, the query will return the complete set of keys for the key-value store.

This is equivalent to self.query(**kwargs).keys(), but potentially more efficiently implemented.

Parameters:kwargs – Arguments where the keywords are metadata keys, and values are possible values for that metadata item.
Returns:result (iterable) - An iterable of key-value store keys whose metadata matches all the specified values for the specified metadata keywords.
set(key, value, buffer_size=1048576)

Store a stream of data into a given key in the key-value store.

This may be left unimplemented by subclasses that represent a read-only key-value store.

Parameters:
  • key (string) – The key for the resource in the key-value store. They key is a unique identifier for the resource within the key-value store.
  • value (instance of Value) – An instance of a Value subclass.
  • buffer_size (int) – An optional indicator of the number of bytes to read at a time. Implementations are free to ignore this hint or use a different default if they need to. The default is 1048576 bytes (1 MiB).
Events:
  • StoreProgressStartEvent - For buffering implementations, this event should be emitted prior to writing any data to the underlying store.
  • StoreProgressStepEvent - For buffering implementations, this event should be emitted periodically as data is written to the underlying store.
  • StoreProgressEndEvent - For buffering implementations, this event should be emitted after finishing writing to the underlying store.
  • StoreSetEvent - On successful completion of a transaction, a StoreSetEvent should be emitted with the key & metadata
set_data(key, data, buffer_size=1048576)

Replace the data for a given key in the key-value store.

Parameters:
  • key (string) – The key for the resource in the key-value store. They key is a unique identifier for the resource within the key-value store.
  • data (file-like) – A readable file-like object the that provides stream of data from the key-value store.
  • buffer_size (int) – An optional indicator of the number of bytes to read at a time. Implementations are free to ignore this hint or use a different default if they need to. The default is 1048576 bytes (1 MiB).
Events:
  • StoreProgressStartEvent - For buffering implementations, this event should be emitted prior to writing any data to the underlying store.
  • StoreProgressStepEvent - For buffering implementations, this event should be emitted periodically as data is written to the underlying store.
  • StoreProgressEndEvent - For buffering implementations, this event should be emitted after finishing writing to the underlying store.
  • StoreSetEvent - On successful completion of a transaction, a StoreSetEvent should be emitted with the key & metadata
set_metadata(key, metadata)

Set new metadata for a given key in the key-value store.

This replaces the existing metadata set for the key with a new set of metadata.

Parameters:
  • key (string) – The key for the resource in the key-value store. They key is a unique identifier for the resource within the key-value store.
  • metadata (dict) – A dictionary of metadata to associate with the key. The dictionary keys should be strings which are valid Python identifiers.
Events:

StoreSetEvent - On successful completion of a transaction, a StoreSetEvent should be emitted with the key & metadata
set_permissions(key, permissions)

Set the permissions on a key the user owns

Parameters:
  • key (str) – The key for the resource which you want to know the permissions.
  • permissions (dict of str: set of str) – A dictionary whose keys are the permissions and values are sets of tags which have that permission. There must be an ‘owned’ permission with at least one tag.
Raises:
  • KeyError - This error will be raised if the key does not exist or the user is not authorized to see it.
  • AuthorizationError - This error will be raised if user is authorized to see the key, but is not an owner.
transaction(notes)

Provide a transaction context manager

This class does not support transactions, so it returns a dummy object.

update_metadata(key, metadata)

Update the metadata for a given key in the key-value store.

This performs a dictionary update on the existing metadata with the provided metadata keys and values

Parameters:
  • key (string) – The key for the resource in the key-value store. They key is a unique identifier for the resource within the key-value store.
  • metadata (dict) – A dictionary of metadata to associate with the key. The dictionary keys should be strings which are valid Python identifiers.
Events:

StoreSetEvent - On successful completion of a transaction, a StoreSetEvent should be emitted with the key & metadata
update_permissions(key, permissions)

Add permissions on a key the user owns

The tags provided in the permissions dictionary will be added to the existing set of tags for each permission.

Parameters:
  • key (str) – The key for the resource which you want to know the permissions.
  • permissions (dict of str: set of str) – A dictionary whose keys are the permissions and values are sets of tags which have that permission.
Raises:
  • KeyError - This error will be raised if the key does not exist or the user is not authorized to see it.
  • AuthorizationError - This error will be raised if user is authorized to see the key, but is not an owner.