Matthias Kruk [Tue, 30 Nov 2021 13:07:29 +0000 (22:07 +0900)]
include/ipc: Don't dereference symlinks when subscribing to a topic
The `_ipc_endpoint_topic_subscribe()' function dereferences symbolic
links when subscribing to a topic, causing symbolic links to be
created in the wrong directory if the topic already exists.
This commit changes the flags that `_ipc_endpoint_topic_subscribe()'
passes to `ln' so that the target won't be dereferenced if it refers
to a symbolic link.
Matthias Kruk [Thu, 25 Nov 2021 05:25:08 +0000 (14:25 +0900)]
Makefile: Disable tests during build process
The test suite is executed automatically by dpkg-buildpackage during
the build process. However, because the test framework is not set up
on the build machine, this causes the packaging process to fail.
This commit removes the test rule from the Makefile to avoid issues
with the build system until the test framework is set up.
Matthias Kruk [Thu, 25 Nov 2021 04:45:42 +0000 (13:45 +0900)]
include/ipc: Add GPLv3 notice
The ipc module does not have a license header, making it hard to tell
what license the code was published under if the file ever gets separated.
This commit adds a license header to the ipc module.
Matthias Kruk [Sun, 8 Aug 2021 06:30:36 +0000 (15:30 +0900)]
include/git: Add module for interaction with git repositories
This commit adds a module that can be used to perform operations such
as cloning, merging, pushing, etc on a git repository without having
to change directories.
include/conf: Add function to get available configuration entries
The conf module currently does not provide a means to figure out what
configuration entries are available.
This commit adds a method that allows the caller to figure out what
configuration settings are available in a given configuration domain.
include/ipc: Add ipc_endpoint_foreach_message() function
The IPC module currently does not provide an API that can be used
to enquire the state of the queue of an endpoint. This makes it
unnecessarily complicated to implement monitoring of IPC endpoints.
This commit adds the ipc_endpoint_foreach_message() function that
may be used to enquire about messages that are stored in the internal
queue of an IPC endpoint. The semantics of this function are similar
to those of queue_foreach(), with the exception that the endpoint
name is passed in the first parameter of the callback and the message
in the second parameter. The remaining parameters are passed on from
ipc_endpoint_foreach_message().
include/conf: Add support for multiple configuration files
The current conf implementation does not permit processes to use
more than one configuration file.
This commit modifies the conf module so that each process may use
an arbitrary number of configuration files by appending the name of
a config domain to the commandline of commands such as conf_get()
and conf_set(), falling back to "default" if the config domain was
omitted.
This commit further adds the conf_get_domains() method that may be
used to query the names of known configuration domains.
Instances don't handle signals, causing them not to get cleaned up
properly when terminated by a signal.
This commit adds a signal handler to the inst module, making sure
instances are properly cleaned up when terminated by a signal.
include/inst: Make subsequent calls to inst_running() return false
When an instance is instructed to stop, a semaphore is increased to
signal this condition. The inst_running() function uses sem_trywait()
on this semaphore, returning false if the semaphore could be passed,
thus telling the caller that the instance should be stopped.
However, since sem_trywait() decreases the semaphore, subsequent calls
to inst_running() will return true again.
This commit changes inst_running() to increase the semaphore in case
it was decreased by sem_trywait(). However, there remains a small
window of opportunity where the function may return different values
to different callers if multiple processes from the same instance call
the function at the same time.
include/inst: Allow functions to be used on foreign instances
Functions of the inst module assume that instances of the current
script are supposed to be manipulated. This makes it impossible for
the functions to be used to enquire or manipulate instances of other
scripts.
This commit changes the functions (where it makes sense) so they can
be used to operate on instances of scripts other than the one that is
executing them.
Several of the test cases for the json module are calling the wrong
functions. Further, there are no test cases for type specifiers in
json_object() and json_array().
This commit fixes test cases that are calling the wrong functions and
adds test cases for type specifiers.
When constructing JSON objects and arrays, there are cases where the
json_object() and json_array() functions cannot correctly determine
the type of a variable (for example, "true" could be string or bool).
This commit introduces type specifiers that allow the user to declare
the type of a value that is passed to the json functions.
Specifiers are prepended to the value when passing it to the
json_object() json_array() functions. The following specifiers are
understood:
For backwards compatibility, the specifiers may be omitted. In that
case, a best-effort guess will be made. Omitting type specifiers will
make it impossible to pass certain values in strings.
include/json: Allow false booleans to be retrieved from JSON types
Because of the way the return value of jq is checked in
json_object_get() and json_array_head(), the functions will fail if a
value of (boolean) "false" was retrieved from the JSON type.
This commit fixes the behavior of json_object_get() and
json_array_head() so that false booleans can be retrieved.
There are issues with bats when accessing device files such as
/dev/urandom, making it hard to implement tests that require randomness.
This commit modifies test.sh to use shellspec as the default test
framework.
To avoid the dependency on ssh, this commit removes the ssh module
from the toolbox core package. The module will be moved to a separate
repository, presumably something along the lines of toolbox-goodies.
The json module doesn't provide any functions to retrieve data from
within JSON types, requiring users to call jq (or other parsers)
directly.
This commit adds the json_object_get() function, which allows the user
to retrieve the value of a field in a JSON object.
Matthias Kruk [Wed, 30 Jun 2021 22:36:10 +0000 (07:36 +0900)]
include/queue: Correctly handle omitted timeout in queue_get_file()
The queue_get_file() function does not correctly interpret the timeout
as infinity if the parameter is omitted.
This commit adds a check to queue_get_file() so that the timeout is
set correctly if the parameter is omitted.
Matthias Kruk [Wed, 30 Jun 2021 22:27:39 +0000 (07:27 +0900)]
include/queue: Don't allow passing data along with files
The queue implementation allows data to be passed along with files in
a queue. This increases the complexity of the queue implementation and
taints the API since the return value from queue_get_file() can't be
used without splitting it first.
This commit removes user-data passing from queue_put_file() and
queue_get_file().
Matthias Kruk [Tue, 29 Jun 2021 00:26:08 +0000 (09:26 +0900)]
test/queue: Add test cases for several queue functions
To ensure that the queue module behaves as intended, test cases are
needed.
This commit adds test cases to test the behavior of the following
functions.
- queue_init()
- queue_destroy()
- queue_put()
- queue_put_unique()
- queue_put_file()
- queue_get()
- queue_foreach()
Matthias Kruk [Mon, 28 Jun 2021 23:28:23 +0000 (08:28 +0900)]
include/queue: Prevent queue_put_unique() from adding duplicates
There are several issues in _queue_contains(), causing the
queue_put_unique() function to hang (waiting for input from stdin) or
add duplicate items to the queue.
This commit addresses these issues so that queue_put_unique() does not
hang or add duplicate items.
Matthias Kruk [Mon, 28 Jun 2021 21:57:38 +0000 (06:57 +0900)]
include/array: Add functions for comparing two arrays
This commit adds the array_same() and array_identical() functions to
the array module. The former allows the caller to determine if two
arrays contain the same elements (i.e. permutations are accepted)
while the latter function does not permit permutations.
To avoid dependencies on packages such as X11 in the toolbox core
package, several packages need to be moved into separate repositories.
This commit removes the following modules from this repository:
- acpi/ac => toolbox-linux
- acpi/battery => toolbox->linux
- clip => toolbox-x11
- gitlab => toolbox-restapis
- iruca => toolbox-restapis
- net/iface => toolbox-linux
- xrandr => toolbox-x11
Matthias Kruk [Sun, 27 Jun 2021 07:35:28 +0000 (16:35 +0900)]
include/queue: Fix missing arrowheads in documentation
The petri net in the comment explaining the queue internals are
missing arrowheads. This may cause confusion since edges in petri
nets are usually directed.
This commit adds the missing arrowheads to the petri net.
Matthias Kruk [Sun, 20 Jun 2021 06:01:45 +0000 (15:01 +0900)]
test/is: Add shellspec test cases for "is" module
The test cases for the "is" module are implemented for BATS, but going
forward shellspec should be used.
This commit migrates the previous test cases from BATS to shellspec and
adds test cases for the is_base64() function.
Matthias Kruk [Sun, 27 Jun 2021 01:35:41 +0000 (10:35 +0900)]
include/ipc: Use endpoints to implement publish-subscribe functions
The publish-subscribe functions added by the previous commit are
independent of the endpoint API, making it necessary to separately
implement authentication and message encapsulation for pubsub
messaging.
This commit moves the pubsub functionality into the endpoint API,
making it unnecessary to re-implement message authentication and
encapsulation.
Matthias Kruk [Sat, 26 Jun 2021 01:58:11 +0000 (10:58 +0900)]
include/log: Fix invalid decision to read from stdin in log_highlight()
Because the boundaries aren't correct, log_highlight() will attempt to
read input from stdin if only one line was passed in positional
parameters.
This commit fixes the boundary so that log_highlight() will print any
number of lines correctly, whether passed in positional parameters or
via stdin.
Matthias Kruk [Thu, 24 Jun 2021 23:46:58 +0000 (08:46 +0900)]
include/log: Remove log_write() from public API
The log_write() function is not intended to be used directly, so it
should not be part of the public API of the log module.
This commit makes the log_write() function private by renaming it to
_log_write() and updating all references to it.
Matthias Kruk [Thu, 24 Jun 2021 23:44:26 +0000 (08:44 +0900)]
include/log: Fix indentation in log_stacktrace()
Stacktraces printed by log_stacktrace() are not indented correctly
because of a typo in the statement that increases the indentation.
This commit fixes the typo so that lines are indented correctly.
Matthias Kruk [Thu, 24 Jun 2021 23:40:01 +0000 (08:40 +0900)]
include: Explicitly declare arguments at the top of all functions
Several functions do not declare and assign all arguments at the top,
violating the toolbox coding style.
This commit changes all argument declarations so that they conform to
the toolbox style.
Matthias Kruk [Mon, 21 Jun 2021 23:49:42 +0000 (08:49 +0900)]
include/{opt,inst}: Add regex parameter to opt_add_arg() calls
The opt and inst modules use opt_add_arg() to register command line
arguments with a callback, however they omit the newly-added regex
argument, causing the callbacks not to be called.
This commit fixes the order of arguments passed to opt_add_arg() so
that the command line arguments are registered correctly.
Matthias Kruk [Mon, 21 Jun 2021 12:13:29 +0000 (21:13 +0900)]
test/ipc: Add test cases for IPC endpoints
Functions that operate on IPC endpoints are not covered by any of the
test cases in the ipc_spec.sh test suite.
This commit adds a set of test cases that test the following functions.
- ipc_endpoint_open()
- ipc_endpoint_close()
- ipc_endpoint_send()
- ipc_endpoint_recv()
Matthias Kruk [Mon, 21 Jun 2021 11:27:57 +0000 (20:27 +0900)]
include/opt: Implement argument validation using regular expressions
The opt_parse() function does not implement validation of commandline
arguments, leaving it to the caller to sanitize values passed by the
user.
This commit adds an argument to opt_add_arg() that allows the caller
to specify a regular expression that will be used by opt_parse() to
validate user inputs.
Matthias Kruk [Fri, 18 Jun 2021 23:52:29 +0000 (08:52 +0900)]
test/ipc: Add test cases for ipc_msg_* functions
This commit adds test cases to ensure that ipc_msg_* functions work as
intended. The following functions are covered by the test cases.
- ipc_msg_get_version()
- ipc_msg_get_source()
- ipc_msg_get_destination()
- ipc_msg_get_user()
- ipc_msg_get_timestamp()
- ipc_msg_get_data()
- ipc_msg_get_signer_name()
- ipc_msg_get_signer_email()
- ipc_msg_get_signer_key()
Matthias Kruk [Fri, 18 Jun 2021 22:55:49 +0000 (07:55 +0900)]
spec/ipc: Rename base message schema to ipc_message / ipc.message.json
IPC base messages are stored in the "message" field of an envelope,
and the schema for the latter is called ipc.envelope.json (and the
file is called ipc_envelope.schema.json). In keeping with that
convention, the schema for IPC base messages should be called
ipc.message.json (filename ipc_message.schema.json).
This commit makes sure the naming conventions are adhered to.
Matthias Kruk [Fri, 18 Jun 2021 07:20:23 +0000 (16:20 +0900)]
include/ipc: Fix invalid function reference
The ipc_endpoint_recv() function calls _ipc_msg_validate(), which has
been renamed to ipc_msg_validate().
This commit fixes the reference so that the correct function is called.
Matthias Kruk [Fri, 18 Jun 2021 07:15:24 +0000 (16:15 +0900)]
include/{mutex,wmutex,queue,sem}: Fix timeout behavior of blocking calls
Because of a bug in the timeout handling, the mutex_lock(),
wmutex_lock(), queue_get(), and sem_wait() functions wait forever if a
positive timeout is specified.
This commit changes the behavior so that the functions will wait
forever if the timeout is -1 or omitted, return immediately if the
timeout is zero, or otherwise wait for the specified number of seconds.
Matthias Kruk [Fri, 18 Jun 2021 00:33:53 +0000 (09:33 +0900)]
test/ipc: Add unit tests for message encoding and authentication
This commit adds unit tests that ensure that the following function
from the ipc module work as intended:
- _ipc_encode()
- _ipc_decode()
- _ipc_sign()
- _ipc_verify()
Matthias Kruk [Thu, 17 Jun 2021 00:09:42 +0000 (09:09 +0900)]
include/ipc: Ensure complete IPC message is signed
The current implementation does not sign fields other than the user
data in IPC messages. This makes it impossible to recognize if the
other fields have been tampered with.
This commit addresses the issue by wrapping the IPC message in an
envelope and putting the signature of the IPC message in the envelope.
This makes sure there is no unsigned data in the communication.
Matthias Kruk [Wed, 16 Jun 2021 23:05:23 +0000 (08:05 +0900)]
include/ipc: Change IPC message format so that all data is signed
In the current IPC message format, only the data field is signed,
causing changes to the other fields to go unnoticed by the signature
verification.
This commit adds a new message format in that the IPC message is
wrapped in an envelope, making sure that all data is authenticated
by the signature.
Matthias Kruk [Wed, 16 Jun 2021 22:02:25 +0000 (07:02 +0900)]
include/ipc: Rename _ipc_msg_encode() and _ipc_msg_decode()
The _ipc_msg_encode() and _ipc_msg_decode() functions are used to
encode and decode more than just IPC messages, so the function names
are somewhat inaccurate.
This commit renames the functions to _ipc_encode() and _ipc_decode(),
respectively.
Matthias Kruk [Tue, 15 Jun 2021 23:23:55 +0000 (08:23 +0900)]
include/ipc: Clean up ipc_msg API
The ipc module currently doesn't provide functions to get all fields
contained within a message. Further, signature-related functions don't
return an error if the message is invalid or the signature could not be
retrieved for another reason.
This commit adds the missing functions to the public API of the ipc
module and fixes the behavior of signature related functions so the
caller can correctly determine if an error has occurred.
The following functions have been added:
- ipc_msg_get_version()
- ipc_msg_get_signature()
- ipc_msg_get_signer_name()
- ipc_msg_get_signer_email()
- ipc_msg_get_signer_key()
Matthias Kruk [Tue, 15 Jun 2021 00:21:48 +0000 (09:21 +0900)]
include/ipc: Simplify IPC endpoint API
The current API of the ipc module expects the user to create an ipc_msg
and send that using the ipc_endpoint_send() function, however there is
nothing useful that can be done with an ipc_msg besides sending it, so
it makes no sense to have the user create the ipc_msg themselves.
Further, the ipc_endpoint_recv() function will return any received
ipc_msg to the user, including invalid or unauthenticated messages.
This commit changes the IPC endpoint API so that the user can pass the
data they intend to send right to ipc_endpoint_send() without having
to call ipc_msg_new(). Since ipc_msg_new() is not intended to be used
by the user directly, it is removed from the public API of the ipc
module.
Further, the ipc_endpoint_recv() function is modified to return only
valid, authenticated messages to the caller.
Matthias Kruk [Mon, 14 Jun 2021 23:31:38 +0000 (08:31 +0900)]
include/ipc: Rename ipc_endpoint_new() to ipc_endpoint_open()
The functions for creating new endpoints and freeing endpoints are
called ipc_endpoint_new() and ipc_endpoint_close(), which is not
intuitive.
This commit renames ipc_endpoint_new() to ipc_endpoint_open(), since
open/close is the terminology that is traditionally used in
communication APIs.
Matthias Kruk [Mon, 14 Jun 2021 22:53:00 +0000 (07:53 +0900)]
include/ipc: Remove option to disable authentication
Providing the option to disable authentication does not bring any real
benefits, while it adds code paths that need to be tested.
This commit removes the option to disable authentication.
Matthias Kruk [Mon, 14 Jun 2021 21:23:02 +0000 (06:23 +0900)]
toolbox: Make have(), _try_include(), include() read-only
Functions from toolbox.sh may be redeclared, causing all sorts of
problems from simple bugs to security issues.
This commit makes sure the functions are declared read-only when
toolbox is loaded so the user can't shoot themself in the foot.
Matthias Kruk [Sun, 13 Jun 2021 23:57:28 +0000 (08:57 +0900)]
include/ipc: Add module for messaging-based IPC
This commit adds the ipc module, which implements a messaging-based
approach to IPC using JSON objects that are transmitted via queues.
The current implementation authenticates messages using GPG, which
allows simple GPG-based authorization schemes to be implemented on
top of this mechanism.
This commit includes a JSON Schema for toolbox IPC messages and a
Python script for validation, both of which will be integrated into
the test suite.
Matthias Kruk [Sun, 13 Jun 2021 08:12:48 +0000 (17:12 +0900)]
include/inst: Add functions for implementing singleton daemons
The inst_start() function does not check if a process is already
running, making it unusable for the implementation of singleton
daemons.
This commit adds the inst_singleton() function, which allows the
caller to implement daemons which can only be run one at a time.
Matthias Kruk [Thu, 10 Jun 2021 20:43:44 +0000 (05:43 +0900)]
COPYING: Add GPLv3 license, copyright headers
None of the source files currently makes any mention of license terms
that toolbox is published under. This commit adds the license text of
the GPLv3 and copyright headers to each of the source files (except
unit tests).
Matthias Kruk [Thu, 10 Jun 2021 05:59:18 +0000 (14:59 +0900)]
include/gitlab: Add functions for interacting with merge requests
To implement a bot that merges merge requests by majority decision,
functions are necessary to get details about merge requests.
This commit adds functions that allow the user to get the number of
members in a project, get details about the votes that have been cast
on a merge requests, add comments to merge requests, and merge merge
requests.
Matthias Kruk [Thu, 10 Jun 2021 00:47:56 +0000 (09:47 +0900)]
include/gitlab: Add function to retrieve all merge requests
This commit adds the gitlab_list_merge_requests() function, which allows
the caller to retrieve a list of merge requests for a given scope. If no
scope is specified, all merge requests that the user is assigned to will
be returned.
Matthias Kruk [Sun, 6 Jun 2021 02:42:13 +0000 (11:42 +0900)]
include/{mutex,wmutex}: Wait forever if timeout is 0 or missing
The mutex_lock() and wmutex_lock() functions return immediately if
the timeout is set to 0 or is missing. However, to be compatible with
the previous timeout-less implementations, the functions must wait
forever if no timeout was specified.
This commit changes the two functions so they wait forever if the
timeout is 0 or missing.
Matthias Kruk [Sun, 6 Jun 2021 02:22:43 +0000 (11:22 +0900)]
include/{mutex,wmutex}: Change mutexes back to the sleep-based approach
Using inotifywait to wait on a mutex does not work reliably when the mutex
is on an NFS and the processes synchronizing on it are running on different
machines.
This commit changes the implementations of mutex and wmutex back to use
sleep instead of inotifywait.
Matthias Kruk [Sun, 6 Jun 2021 02:08:00 +0000 (11:08 +0900)]
include/inst: Use new opt flags
Calls to opt_add_arg() in the inst module still use the old way to
specify option flags, which is incompatible with the current opt
module.
This commit changes the calls to pass the option flags in the new way.
Matthias Kruk [Sun, 6 Jun 2021 02:03:56 +0000 (11:03 +0900)]
include/inst: Implement instance status messages
In order to make the status of an instance visible to users and other
processes, functions are needed that can be used to set status messages
from within an instance.
This commit adds status messages for instances, which can be manipulated
through the following functions.
- inst_set_status(): Set the status of the running instance
- inst_get_status_message(): The the status message of an instance
- inst_get_status_timestamp(): Get the timestamp of a status message
- inst_get_status(): Get status message and timestamp of an instance
Matthias Kruk [Sat, 29 May 2021 04:17:48 +0000 (13:17 +0900)]
include/{mutex,wmutex,sem,queue}: Add timeout argument to blocking functions
The following functions potentially block the executing script indefinitely,
without a way to recover from the situation.
- queue_get()
- queue_get_file()
- sem_wait()
- mutex_lock()
- wmutex_lock()
It is a common pattern to implement deamons dispatching messages from a queue
in the following way using the "inst" module.
while inst_running; do
local msg
if msg=$(queue_get "$queue"); then
handle_message "$msg"
fi
done
The problem with this code is that queue_get() potentially waits indefinitely,
and the daemon won't terminate even if it was instructed to. To mitigate this
situation, this commit adds a timeout argument to all of the above functions,
allowing the caller to specify the number of seconds the functions should wait
at most. If the timeout parameter is 0 or omitted, the functions will wait
forever (as is the current behavior), so existing code that uses any of the
above functions and that wishes to retain the current behavior does not need to
be modified.
Matthias Kruk [Fri, 28 May 2021 12:12:16 +0000 (21:12 +0900)]
include/json: Add functions for manipulating JSON arrays
Handling JSON arrays in shell scripts can be painful without jq. This
commit adds a number of convenience functions that wrap jq to make
array handling easier. The follow fuctions are added:
- json_array_head(): Returns the first element of a JSON array
- json_array_tail(): Returns a new JSON array without the first element
- json_array_to_lines(): Returns all elements of a JSON array, one per line
Matthias Kruk [Wed, 26 May 2021 04:31:15 +0000 (13:31 +0900)]
include/queue: queue_foreach: Return failure if callback fails
The queue_foreach() function always returns success, even if one of
the callbacks failed. This makes it hard to handle errors that occur
inside the callbacks.
This commit changes the function to return failure if a callback has
failed.
Matthias Kruk [Fri, 21 May 2021 07:04:45 +0000 (16:04 +0900)]
include/json: Correctly escape braces in RegEx
The json functions currently don't correctly recognize objects because
braces aren't correctly escaped. This commit changes the regexes so that
braces are escaped properly.
Matthias Kruk [Fri, 21 May 2021 06:39:19 +0000 (15:39 +0900)]
include/json: Allow arrays, objects as input to json_array() and json_object()
The json_array() and json_object() functions currently can only be used
to construct arrays and objects containing strings and integers.
This commit modifies the functions so that they can also encode objects
and arrays within objects and arrays.
Matthias Kruk [Thu, 20 May 2021 04:37:59 +0000 (13:37 +0900)]
include/log: Don't pass log message through date
The log_write() function passes the log message through `date' to
prepend the timestamp to it. This causes problems when the message
contains format placeholders that are interpreted by date.
This commit changes the log_write() function not to pass the log
message and prefix through date.
Matthias Kruk [Thu, 20 May 2021 00:58:31 +0000 (09:58 +0900)]
include/opt: Fix C-ish statement
The statement used to store the length of the longest option name in
opt_add_arg() looks very C-like, meaning it doesn't work correctly in
bash. This commit changes the statement to proper shell syntax.
Matthias Kruk [Thu, 20 May 2021 00:55:14 +0000 (09:55 +0900)]
include/opt: Work around bug in old bash versions
Declaring a non-local array and initializing it in the same statement
causes it to become local in older bash versions. This commit works
around this bug by removing the array initializations from internal
arrays of the opt module.
Matthias Kruk [Wed, 19 May 2021 23:32:28 +0000 (08:32 +0900)]
include/gitlab: Allow destination namespace to be passed to fork methods
The gitlab_fork() and gitlab_fork_sync() functions can only be used to
for a project to the user's personal namespace, since they do not allow
the destination namespace to be specified. This commit adds an argument
for the destination namespace, so that the functions can be used to fork
into arbitrary namespaces.
Matthias Kruk [Wed, 19 May 2021 23:26:24 +0000 (08:26 +0900)]
include/gitlab: Add method to query the current user
This commit adds the gitlab_get_current_user() function, which allows
the caller to retrieve a JSON object containing information about the
current GitLab user.
include/queue: Allow additional data to be passed along with files
Some use-cases require files to be passed through queues along with
additional data. This commit extends the queue_put_file() and
queue_get_file() functions to allow additional data to be passed
along with a file.
There is currently no way to inspect the state of a queue, making it
very hard to write scripts that safely show the contents of a queue.
This commit adds the queue_foreach() function, which allows the caller
to invoke a callback function for each item in a queue.
The queue_foreach() function must be passed at least two arguments,
the name of the queue and the name of the callback function. All other
arguments will be passed on to the callback function.
If the callback returns a non-zero value, the iteration of the queue
will be aborted.
The queue_foreach() function will lock the queue for the duration that
the callbacks are executed. The callbacks must under absolutely no
circumstances attempt to perform any queue operations on the queue or
otherwise attempt to modify the queue, since this will either result
in a deadlock or leave the queue in an inconsistent state.
Example use:
callback() {
local item="$1"
local user_data="$2"
local more_data="$3"
include/sem: Re-implement semaphores without busy-waiting workaround
The current sem module has a design issue that requires the use of sleep
or inotifywait to avoid busy-waiting when the semaphore counter is zero.
This commit adds a new sem implementation that avoids the problem
altogether through the use of a separate mutex (called waitlock), which
sem_wait() will attempt to acquire, causing it to be blocked until the
waitlock is released by a call to sem_post().
Further, the new implementation keeps all filesystem objects belonging to
a semaphore in a directory, giving it a cleaner appearance.
This commit also adds several test cases for the new sem module.
include/mutex,wmutex: Add weak mutexes that don't enforce ownership
When using mutexes as a means for signalling between two processes, the
lock and unlock operation will almost never be executed by the same
process. However, the mutex module does not allow such behavior.
This commit adds the wmutex module (the "w" is for "weak") which does
not enforce ownership on mutexes.
As a side note, the same could have been achieved by adding something
like mutex_weak_lock() and mutex_weak_unlock() to the mutex module, but
that would lead to people mixing the two. Implementing them in separate
modules hopefully makes it more clear that using wmutex functions on a
mutex (and vice-versa) is not acceptable. If necessary, I might even
implement the two in different ways so a mutex can't be released with
wmutex_unlock().
include/conf: Append config instead of overwriting it
The conf_set() function overwrites the configuration file instead of
appending to it, making it impossible to store multiple settings.
This commit changes conf_set() to append to the configuration file.
include/mutex: Prevent children from unlocking their parents' mutexes
The mutex_trylock() and mutex_unlock() functions use $$ to determine the
current process's PID. However, this variable always reflects the PID of
the original shell. This commit changes the functions to use $BASHPID to
determine the PID of the current shell.
include/mutex: Make mutex_lock() fail if the path does not exist
The mutex_lock() function does not check if the path of the mutex
exists, causing it to wait forever if an invalid path was passed.
This commit makes mutex_lock() fail if it could not inotifywait on
the directory that contains the mutex.
The is_alpha() and is_alnum() functions fail to recognize strings
containing mixed case and digits (in the case of the latter). This
commit fixes the implementations by making perform the check with a
regular expression instead of delegating the work to is_upper(),
is_lower(), and is_digits().
The name of the is_digit() function was originally taken from the
Standard C library function of the same name. However, that function
checks only a single character, while this function compares all
characters in a string. To avoid confusion, this commit renames the
function to is_digits().
It is more common to use -q or --quiet to make commandline utilities
emit less output. This commit renames the -w/--shush argument to -q/
--quiet to follow that convention.
include/opt: Add flag to define required arguments
The opt module currently does not provide a means to define required
arguments, making it hard for the callee to determine if an argument
was provided on the commandline, or if the default value was returned
by opt_get().
This commit adds a flag to specify if an argument must be provided on
the commandline. Further, since multiple flags can now be passed to
opt_add_arg(), the flag format is changed: now, a string containing
all flags as characters must be passed. The implementation understands
the following flags:
r - Argument is required
v - Argument has a value (e.g. is followed by a second argument)
For example, to declare a required argument with a value, one would
pass the flags "rv".
This commit increases the package version to 0.3-1 and removes the
description from the "Source" portion of the debian/control file so
the dpkg-buildpackage version used on deb.m10k.eu won't complain.
The queue module depends on the log module, however it doesn't include
it during initialization. This commit makes sure the log module is
loaded along with the queue module.
include/sem: Work around busy-waiting in sem_wait()
The sem_wait() function is implemented using busy-waiting, which is not
good for the environment. This commit adds a call to inotifywait as a
temporary workaround to avoid the busy-waiting.
include/mutex: Use inotifywait to improve performance
The current mutex implementation uses sleep to periodically check if
the lock can be acquired. This causes a lot of latency for short waits
and a lot of unnecessary sleep/wakeup cycles for long waits.
This commit changes the mutex implementation to use inotifywait
instead of sleep, increasing mutex performance for both short and long
waits.
include/inst: Make inst_stop emit a meaningful message upon error
Instead of emitting a user-understandable error message, the inst_stop()
function merely lets an error message from the sem module slip through.
This commit changes the function to emit an error message that tells the
caller why exactly the function failed.
include/inst: Add inst module for managing running instances of scripts
This commit adds the inst module which provides functions for managing
instances of scripts:
- The inst_start() function forks a function to the background
- The inst_stop() function asks an instance to stop
- The inst_list() function lists running instances
- The inst_running() function tells an instance if it should stop
Internally, the module makes use of semaphores to list instances and
determine their state. Semaphores are kept in per-script directories
within $TOOLBOX_HOME/inst.
include/opt: Remember arguments passed to opt_parse()
To allow the user to distinguish running instances of a script, it is
necessary to present the arguments that an instance was started with.
The most logical place to remember such information is inside the opt
module. Therefore, this commit adds the opt_get_argv() function, which
allows the caller to retrieve the arguments that were passed to the
opt_parse() function.
include/opt: Make --verbose and --shush default arguments
The -v/--verbose and -w/--shush options are frequently implemented in
scripts that use toolbox. To reduce code duplication and to make all
scripts follow the same argument naming conventions, this commit makes
-v/--verbose and -w/--shush standard arguments of the opt module.
include/log: Add functions to increase and decrease the verbosity
I frequently find myself implementing the same functions to increase
and decrease the verbosity of my scripts. This commit adds these
functions to the log module.
include/queue: Add functions for file queues and duplicate-free queues
For certain automation tasks, it would be helpful if filesystem objects
could be passed through queues. This commit adds such functionality to
the queue module.
For passing files, these functions can be used:
- queue_put_file()
- queue_get_file()
For duplicate-free transient data:
- queue_put_unique()
- queue_get()
For other transient data:
- queue_put()
- queue_get()
There are two things worth noting:
1. The queues are line-based (transient data may not contain newlines)
2. Queues cannot be used for transient data and files at the same time