7. Stores

A store is a fixed-size block of storage, which can be read and perhaps written to. A store is more general than a file: it refers to any type of storage such as devices, files, memory, tasks, etc. Stores can also be representations of other stores, which may be combined and filtered in various ways.

7.1 storeinfo, storecat, storeread

7.2 storeio

FIXME: finish

7.3 Store Library

The store library (which is declared in <hurd/store.h>) implements many different backends which support the store abstraction. Hurd programs use libstore so that new storage types can be implemented with minimum impact.

7.3.1 Store Arguments

FIXME: describe startup sequence

Structure: struct store_parsed

The result of parsing a store, which should be enough information to open it, or return the arguments.

Structure: struct store_argp_params { struct store_parsed *result; const char *default_type; const struct store_class *const *classes; }

This is the structure used to pass args back and forth from store_argp. result is the resulting parsed result. If `--store-type' isn't specified, then default_type should be used as the store type; zero is equivalent to "query". classes is set of classes used to validate store types and argument syntax.

Variable: extern struct argp store_argp

This is an argument parser that may be used for parsing a simple command line specification for stores. The accompanying input parameter must be a pointer to a struct store_argp_params.

Function: void store_parsed_free (struct store_parsed *parsed)

Free all resources used by parsed.

Function: error_t store_parsed_open (const struct store_parsed *parsed, int flags, struct store **store)

Open the store specified by parsed, and return it in store.

Function: error_t store_parsed_append_args (const struct store_parsed *parsed, char **argz, size_t *argz_len)

Add the arguments used to create parsed to argz and argz_len.

Function: error_t store_parsed_name (const struct store_parsed *parsed, char **name)

Make an option string describing parsed, and return it in malloced storage in name.

7.3.2 Store Management

The following functions provide basic management of stores:

Function: error_t store_create (file_t source, int flags, const struct store_class *const *classes, struct store **store)

Return a new store in store, which refers to the storage underlying source. classes is used to select classes specified by the provider; if zero, store_std_classes is used. flags is set with store_set_flags, with the exception of STORE_INACTIVE, which merely indicates that no attempt should be made to activate an inactive store; if STORE_INACTIVE is not specified, and the store returned for SOURCE is inactive, an attempt is made to activate it (failure of which causes an error to be returned). A reference to source is created (but may be destroyed with store_close_source).

It is usually better to use a specific store open or create function such as store_open (see section 7.3.4 Store Classes), since they are tailored to the needs of a specific store. Generally, you should only use store_create if you are defining your own store class, or you need options that are not provided by a more specific store creation function.

Function: void store_close_source (struct store *store)

If store was created using store_create, remove the reference to the source from which it was created.

Function: void store_free (struct store *store)

Clean up and deallocate store's underlying stores.

Structure: struct store_run { store_offset_t start, length; }

A struct store_run represents a contiguous region in a store's address range. These are used to designate active portions of a store. If start is -1, then the region is a hole (it is zero-filled and doesn't correspond to any real addresses).

Function: error_t store_set_runs (struct store *store, const struct store_run *runs, size_t num_runs)

Set store's current runs list to (a copy of) runs and num_runs.

Function: error_t store_set_children (struct store *store, struct store *const *children, size_t num_children)

Set store's current children to (a copy of) children and num_children (note that just the vector children is copied, not the actual children).

Function: error_t store_children_name (const struct store *store, char **name)

Try to come up with a name for the children in store, combining the names of each child in a way that could be used to parse them with store_open_children. This is done heuristically, and so may not succeed. If a child doesn't have a name, EINVAL is returned.

Function: error_t store_set_name (struct store *store, const char *name)

Sets the name associated with store to a copy of name.

Function: error_t store_set_flags (struct store *store, int flags)

Add flags to store's currently set flags.

Function: error_t store_clear_flags (struct store *store, int flags)

Remove flags from store's currently set flags.

Function: error_t store_set_child_flags (struct store *store, int flags)

Set flags in all children of store, and if successful, add flags to store's flags.

Function: error_t store_clear_child_flags (struct store *store, int flags)

Clear flags in all children of store, and if successful, remove flags from store's flags.

Function: int store_is_securely_returnable (struct store *store, int open_flags)

Returns true if store can safely be returned to a user who has accessed it via a node using open_flags, without compromising security.

Function: error_t store_clone (struct store *from, struct store **to)

Return a copy of from in to.

Function: error_t store_remap (struct store *source, const struct store_run *runs, size_t num_runs, struct store **store)

Return a store in store that reflects the blocks in runs and runs_len from source; source is consumed, but not runs. Unlike the store_remap_create function, this may simply modify source and return it.

7.3.3 Store I/O

The following functions allow you to read and modify the contents of a store:

Function: error_t store_map (const struct store *store, vm_prot_t prot, mach_port_t *memobj)

Return a memory object paging on store.

Function: error_t store_read (struct store *store, store_offset_t addr, size_t amount, void **buf, size_t *len)

Read amount bytes from store at addr into buf and len (which follows the usual Mach buffer-return semantics) to store at addr. addr is in blocks (as defined by store->block_size). Note that len is in bytes.

Function: error_t store_write (struct store *store, store_offset_t addr, void *buf, size_t len, size_t *amount)

Write len bytes from buf to store at addr. Returns the amount written in amount (in bytes). addr is in blocks (as defined by store->block_size).

Function: error_t store_set_size (struct store *store, store_offset_t newsize)

Set store's size to newsize (in bytes).

7.3.4 Store Classes

The store library comes with a number of standard store class implementations:

Variable: extern const struct store_class *const store_std_classes []

This is a null-terminated vector of the standard store classes implemented by libstore.

If you are building your own class vectors, the following function may be useful:

Variable: error_t store_concat_class_vectors (struct store_class **cv1, struct store_class **cv2, struct store_class ***concat)

Concatenate the store class vectors in cv1 and cv2, and return a new (malloced) vector in concat.

7.3.4.1 query store

Variable: extern const struct store_class store_query_class

This store is a virtual store which queries a filesystem node, and delegates control to an appropriate store class.

Function: error_t store_open (const char *name, int flags, const struct store_class *const *classes, struct store **store)

Open the file name, and return a new store in store, which refers to the storage underlying it. classes is used to select classes specified by the provider; if it is zero, then store_std_classes is used. flags is set with store_set_flags. A reference to the open file is created (but may be destroyed with store_close_source).

7.3.4.2 typed_open store

Variable: extern const struct store_class store_typed_open_class

This store is special in that it doesn't correspond to any specific store functions, rather it provides a way to interpret character strings as specifications for other stores.

Function: error_t store_typed_open (const char *name, int flags, const struct store_class *const *classes, struct store **store)

Open the store indicated by name, which should consist of a store type name followed by a `:' and any type-specific name, returning the new store in store. classes is used to select classes specified by the type name; if it is zero, store_std_classes is used.

Function: error_t store_open_children (const char *name, int flags, const struct store_class *const *classes, struct store ***stores, size_t *num_stores)

Parse multiple store names in name, and open each individually, returning all in the vector stores, and the number in num_stores. The syntax of name is a single non-alphanumeric separator character, followed by each child store name separated by the same separator; each child name is `type:name' notation as parsed by store_typed_open. If every child uses the same `type:' prefix, then it may be factored out and put before the child list instead (the two notations are differentiated by whether or not the first character of name is alphanumeric).

7.3.4.3 device store

Variable: extern const struct store_class store_device_class

This store is a simple wrapper for a microkernel device driver.(7)

Function: error_t store_device_open (const char *name, int flags, struct store **store)

Open the device named name, and return the corresponding store in store.

Function: error_t store_device_create (device_t device, int flags, struct store **store)

Return a new store in store referring to the microkernel device device. Consumes the device send right.

7.3.4.4 file store

Variable: extern const struct store_class store_file_class

This store reads and writes the contents of a Hurd file.

Function: error_t store_file_open (const char *name, int flags, struct store **store)

Open the file name, and return the corresponding store in store.

Function: error_t store_file_create (file_t file, int flags, struct store **store)

Return a new store in store referring to the file file. Unlike store_create, this will always use file I/O, even it would be possible to be more direct. This may work in more cases, for instance if the file has holes. Consumes the file send right.

7.3.4.5 task store

Variable: extern const struct store_class store_task_class

This store provides access to the contents of a microkernel task.

Variable: error_t store_task_open (const char *name, int flags, struct store **store)

Open the task name (name should be the task's pid), and return the corresponding store in store.

Variable: error_t store_task_create (task_t task, int flags, struct store **store)

Return a new store in store referring to the task task, consuming the task send right.

7.3.4.6 zero store

Variable: extern const struct store_class store_zero_class

Reads to this store always return zero-filled buffers, no matter what has been written into it. This store corresponds to the Unix `/dev/zero' device node.

Function: error_t store_zero_create (store_offset_t size, int flags, struct store **store)

Return a new zero store size bytes long in store.

7.3.4.7 copy store

Variable: extern const struct store_class store_copy_class

This store provides a temporary copy of another store. This is useful if you want to provide writable data, but do not wish to modify the underlying store. All changes to a copy store are lost when it is closed.

Function: error_t store_copy_open (const char *name, int flags, const struct store_class *const *classes, struct store **store)

Open the copy store name (which consists of another store class name, a `:', and a name for the store class to open) and return the corresponding store in store. classes is used to select classes specified by the type name; if it is zero, store_std_classes is used.

Function: error_t store_copy_create (struct store *from, int flags, struct store **store)

Return a new store in store which contains a snapshot of the contents of the store from; from is consumed.

Function: error_t store_buffer_create (void *buf, size_t buf_len, int flags, struct store **store)

Return a new store in store which contains the memory buffer buf, of length buf_len. buf must be allocated with vm_allocate, and will be consumed.

7.3.4.8 gunzip store

Variable: extern const struct store_class store_gunzip_class

This store provides transparent GNU zip decompression of a substore. Unfortunately, this store is currently read-only.

Variable: error_t store_gunzip_open (const char *name, int flags, const struct store_class *const *classes, struct store **store)

Open the gunzip store name (which consists of another store class name, a `:', and a name for that store class to open), and return the corresponding store in store. classes is used to select classes specified by the type name; if it is zero, store_std_classes is used.

Variable: error_t store_gunzip_create (struct store *from, int flags, struct store **store)

Return a new store in store which contains a snapshot of the uncompressed contents of the store from; from is consumed. block_size is the desired block size of the result.

7.3.4.9 concat store

Variable: extern const struct store_class store_concat_class

This class provides a linear concatenation storage mode. It creates a new virtual store which consists of several different substores appended to one another.

This mode is designed to increase storage capacity, so that when one substore is filled, new data is transparently written to the next substore. Concatenation requires robust hardware, since a failure in any single substore will wipe out a large section of the data.

Function: error_t store_concat_open (const char *name, int flags, const struct store_class *const *classes, struct store **store)

Return a new store that concatenates the stores created by opening all the individual stores described in name; for the syntax of name, see store_open_children.

Function: error_t store_concat_create (struct store * const *stores, size_t num_stores, int flags, struct store **store)

Return a new store in store that concatenates all the stores in stores (num_stores of them). The stores in stores are consumed; that is, they will be freed when this store is freed. The stores array, however, is copied, and so should be freed by the caller.

7.3.4.10 ileave store

Variable: extern const struct store_class store_ileave_class

This class provides a RAID-0(8) storage mode (also called disk striping). It creates a new virtual store by interleaving the contents of several different substores.

This RAID mode is designed to increase storage performance, since I/O will probably occur in parallel if the substores reside on different physical devices. Interleaving works best with evenly-yoked substores... if the stores are different sizes, some space will be not be used at the end of the larger stores; if the stores are different speeds, then I/O will have to wait for the slowest store; if some stores are not as reliable as others, failures will wipe out every nth storage block, where n is the number of substores.

Function: error_t store_ileave_create (struct store * const *stripes, size_t num_stripes, store_offset_t interleave, int flags, struct store **store)

Return a new store in store that interleaves all the stores in stripes (num_stripes of them) every interleave bytes; interleave must be an integer multiple of each stripe's block size. The stores in stripes are consumed; that is, they will be freed when this store is freed. The stripes array, however, is copied, and so should be freed by the caller.

7.3.4.11 mvol store

Variable: extern const struct store_class store_mvol_class

This store provides access to multiple volumes using a single-volume device. One use of this store would be to provide a store which consists of multiple floppy disks when there is only a single disk drive. It works by remapping a single linear address range to multiple address ranges, and keeping track of the currently active range. Whenever a request maps to a range that is not active, a callback is made in order to switch to the new range.

This class is not included in store_std_classes, because it requires an application-specific callback.

Function: error_t store_mvol_create (struct store *phys, error_t (*swap_vols) (struct store *store, size_t new_vol, ssize_t old_vol), int flags, struct store **store)

Return a new store in store that multiplexes multiple physical volumes from phys as one larger virtual volume. swap_vols is a function that will be called whenever reads or writes refer to a block which is not addressable on the currently active volume. phys is consumed.

7.3.4.12 remap store

Variable: extern const struct store_class store_remap_class

This store translates I/O requests into different addresses on a different store.

Function: error_t store_remap_create (struct store *source, const struct store_run *runs, size_t num_runs, int flags, struct store **store)

Return a new store in store that reflects the blocks in runs and runs_len from source; source is consumed, but runs is not. Unlike the store_remap function, this function always operates by creating a new store of type `remap' which has source as a child, and so may be less efficient than store_remap for some types of stores.

7.3.5 Store RPC Encoding

The store library also provides some functions which help transfer stores between tasks via RPC:

Structure: struct store_enc

This structure is used to hold the various bits that make up the representation of a store for transmission via RPC. See <hurd/hurd_types.h> for an explanation of the encodings for the various storage types.

Function: void store_enc_init (struct store_enc *enc, mach_port_t *ports, mach_msg_type_number_t num_ports, int *ints, mach_msg_type_number_t num_ints, off_t *offsets, mach_msg_type_number_t num_offsets, char *data, mach_msg_type_number_t data_len)

Initialize enc. The given vector and sizes will be used for the encoding if they are big enough (otherwise new ones will be automatically allocated).

Function: void store_enc_dealloc (struct store_enc *enc)

Deallocate storage used by the fields in enc (but nothing is done with enc itself).

Function: void store_enc_return (struct store_enc *enc, mach_port_t **ports, mach_msg_type_number_t *num_ports, int **ints, mach_msg_type_number_t *num_ints, off_t **offsets, mach_msg_type_number_t *num_offsets, char **data, mach_msg_type_number_t *data_len)

Copy out the parameters from enc into the given variables suitably for returning from a file_get_storage_info RPC, and deallocate enc.

Function: error_t store_return (const struct store *store, mach_port_t **ports, mach_msg_type_number_t *num_ports, int **ints, mach_msg_type_number_t *num_ints, off_t **offsets, mach_msg_type_number_t *num_offsets, char **data, mach_msg_type_number_t *data_len)

Encode store into the given return variables, suitably for returning from a file_get_storage_info RPC.

Function: error_t store_encode (const struct store *store, struct store_enc *enc)

Encode store into enc, which should have been prepared with store_enc_init, or return an error. The contents of enc may then be returned as the value of file_get_storage_info; if for some reason this can't be done, store_enc_dealloc may be used to deallocate the memory used by the unsent vectors.

Function: error_t store_decode (struct store_enc *enc, const struct store_class *const *classes, struct store **store)

Decode enc, either returning a new store in store, or an error. classes is the mapping from Hurd storage class ids to store classes; if it is zero, store_std_classes is used. If nothing else is to be done with enc, its contents may then be freed using store_enc_dealloc.

Function: error_t store_allocate_child_encodings (const struct store *store, struct store_enc *enc)

Calls the allocate_encoding method in each child store of store, propagating any errors. If any child does not have such a method, EOPNOTSUPP is returned.

Function: error_t store_encode_children (const struct store *store, struct store_enc *enc)

Calls the encode method in each child store of store, propagating any errors. If any child does not have such a method, EOPNOTSUPP is returned.

Function: error_t store_decode_children (struct store_enc *enc, int num_children, const struct store_class *const *classes, struct store **children)

Decodes num_children from enc, storing the results into successive positions in children.

Function: error_t store_with_decoded_runs (struct store_enc *enc, size_t num_runs, error_t (*fun) (const struct store_run *runs, size_t num_runs))

Call fun with the vector runs of length num_runs extracted from enc.

Function: error_t store_std_leaf_allocate_encoding (const struct store *store, struct store_enc *enc)

Function: error_t store_std_leaf_encode (const struct store *store, struct store_enc *enc)

Standard encoding used for most data-providing (as opposed to filtering) store classes.

Typedef: typedef error_t (*store_std_leaf_create_t) (mach_port_t port, int flags, size_t block_size, const struct store_run *runs, size_t num_runs, struct store **store)

Creation function used by store_std_leaf_decode.

Function: error_t store_std_leaf_decode (struct store_enc *enc, store_std_leaf_create_t create, struct store **store)

Decodes the standard leaf encoding which is common to various builtin formats, and calls create to actually create the store.