sorcery/res_pjsip: Refactor for realtime performance

There were a number of places in the res_pjsip stack that were getting
all endpoints or all aors, and then filtering them locally.

A good example is pjsip_options which, on startup, retrieves all
endpoints, then the aors for those endpoints, then tests the aors to see
if the qualify_frequency is > 0.  One issue was that it never did
anything with the endpoints other than retrieve the aors so we probably
could have skipped a step and just retrieved all aors. But nevermind.

This worked reasonably well with local config files but with a realtime
backend and thousands of objects, this was a nightmare.  The issue
really boiled down to the fact that while realtime supports predicates
that are passed to the database engine, the non-realtime sorcery
backends didn't.

They do now.

The realtime engines have a scheme for doing simple comparisons. They
take in an ast_variable (or list) for matching, and the name of each
variable can contain an operator.  For instance, a name of
"qualify_frequency >" and a value of "0" would create a SQL predicate
that looks like "where qualify_frequency > '0'".  If there's no operator
after the name, the engines add an '=' so a simple name of
"qualify_frequency" and a value of "10" would return exact matches.

The non-realtime backends decide whether to include an object in a
result set by calling ast_sorcery_changeset_create on every object in
the internal container.  However, ast_sorcery_changeset_create only does
exact string matches though so a name of "qualify_frequency >" and a
value of "0" returns nothing because the literal "qualify_frequency >"
doesn't match any name in the objset set.

So, the real task was to create a generic string matcher that can take a
left value, operator and a right value and perform the match. To that
end, strings.c has a new ast_strings_match(left, operator, right)
function.  Left and right are the strings to operate on and the operator
can be a string containing any of the following: = (or NULL or ""), !=,
>, >=, <, <=, like or regex.  If the operator is like or regex, the
right string should be a %-pattern or a regex expression.  If both left
and right can be converted to float, then a numeric comparison is
performed, otherwise a string comparison is performed.

To use this new function on ast_variables, 2 new functions were added to
config.c.  One that compares 2 ast_variables, and one that compares 2
ast_variable lists.  The former is useful when you want to compare 2
ast_variables that happen to be in a list but don't want to traverse the
list.  The latter will traverse the right list and return true if all
the variables in it match the left list.

Now, the backends' fields_cmp functions call ast_variable_lists_match
instead of ast_sorcery_changeset_create and they can now process the
same syntax as the realtime engines.  The realtime backend just passes
the variable list unaltered to the engine.  The only gotcha is that
there's no common realtime engine support for regex so that's been noted
in the api docs for ast_sorcery_retrieve_by_fields.

Only one more change to sorcery was done...  A new config flag
"allow_unqualified_fetch" was added to reg_sorcery_realtime.
"no": ignore fetches if no predicate fields were supplied.
"error": same as no but emit an error. (good for testing)
"yes": allow (the default);
"warn": allow but emit a warning. (good for testing)

Now on to res_pjsip...

pjsip_options was modified to retrieve aors with qualify_frequency > 0
rather than all endpoints then all aors.  Not only was this a big
improvement in realtime retrieval but even for config files there's an
improvement because we're not going through endpoints anymore.

res_pjsip_mwi was modified to retieve only endpoints with something in
the mailboxes field instead of all endpoints then testing mailboxes.

res_pjsip_registrar_expire was completely refactored.  It was retrieving
all contacts then setting up scheduler entries to check for expiration.
Now, it's a single thread (like keepalive) that periodically retrieves
only contacts whose expiration time is < now and deletes them.  A new
contact_expiration_check_interval was added to global with a default of
30 seconds.

Ross Beer reports that with this patch, his Asterisk startup time dropped
from around an hour to under 30 seconds.

There are still objects that can't be filtered at the database like
identifies, transports, and registrations.  These are not going to be
anywhere near as numerous as endpoints, aors, auths, contacts however.

Back to allow_unqualified_fetch.  If this is set to yes and you have a
very large number of objects in the database, the pjsip CLI commands
will attempt to retrive ALL of them if not qualified with a LIKE.
Worse, if you type "pjsip show endpoint <tab>" guess what's going to
happen? :)  Having a cache helps but all the objects will have to be
retrieved at least once to fill the cache.  Setting
allow_unqualified_fetch=no prevents the mass retrieve and should be used
on endpoints, auths, aors, and contacts.  It should NOT be used for
identifies, registrations and transports since these MUST be
retrieved in bulk.

Example sorcery.conf:

[res_pjsip]
endpoint=config,pjsip.conf,criteria=type=endpoint
endpoint=realtime,ps_endpoints,allow_unqualified_fetch=error

ASTERISK-25826 #close
Reported-by: Ross Beer
Tested-by: Ross Beer

Change-Id: Id2691e447db90892890036e663aaf907b2dc1c67

10 files changed, 222 insertions, 351 deletions

diff --git a/res/res_pjsip.c b/res/res_pjsip.c
index 170a19151..67d8dce38 100644
--- a/res/res_pjsip.c
+++ b/res/res_pjsip.c
@@ -1282,6 +1282,9 @@
 				<configOption name="keep_alive_interval" default="0">
 					<synopsis>The interval (in seconds) to send keepalives to active connection-oriented transports.</synopsis>
 				</configOption>
+				<configOption name="contact_expiration_check_interval" default="30">
+					<synopsis>The interval (in seconds) to check for expired contacts.</synopsis>
+				</configOption>
 				<configOption name="max_initial_qualify_time" default="0">
 					<synopsis>The maximum amount of time from startup that qualifies should be attempted on all contacts.
 					If greater than the qualify_frequency for an aor, qualify_frequency will be used instead.</synopsis>
diff --git a/res/res_pjsip/config_global.c b/res/res_pjsip/config_global.c
index 3d88ffc2a..c0fede64d 100644
--- a/res/res_pjsip/config_global.c
+++ b/res/res_pjsip/config_global.c
@@ -36,6 +36,7 @@
 #define DEFAULT_MAX_INITIAL_QUALIFY_TIME 0
 #define DEFAULT_FROM_USER "asterisk"
 #define DEFAULT_REGCONTEXT ""
+#define DEFAULT_CONTACT_EXPIRATION_CHECK_INTERVAL 30
 
 static char default_useragent[256];
 
@@ -58,6 +59,8 @@ struct global_config {
 	unsigned int keep_alive_interval;
 	/* The maximum time for all contacts to be qualified at startup */
 	unsigned int max_initial_qualify_time;
+	/* The interval at which to check for expired contacts */
+	unsigned int contact_expiration_check_interval;
 };
 
 static void global_destructor(void *obj)
@@ -186,6 +189,21 @@ unsigned int ast_sip_get_keep_alive_interval(void)
 	return interval;
 }
 
+unsigned int ast_sip_get_contact_expiration_check_interval(void)
+{
+	unsigned int interval;
+	struct global_config *cfg;
+
+	cfg = get_global_cfg();
+	if (!cfg) {
+		return DEFAULT_CONTACT_EXPIRATION_CHECK_INTERVAL;
+	}
+
+	interval = cfg->contact_expiration_check_interval;
+	ao2_ref(cfg, -1);
+	return interval;
+}
+
 unsigned int ast_sip_get_max_initial_qualify_time(void)
 {
 	unsigned int time;
@@ -331,6 +349,9 @@ int ast_sip_initialize_sorcery_global(void)
 		OPT_STRINGFIELD_T, 0, STRFLDSET(struct global_config, default_from_user));
 	ast_sorcery_object_field_register(sorcery, "global", "regcontext", DEFAULT_REGCONTEXT,
                 OPT_STRINGFIELD_T, 0, STRFLDSET(struct global_config, regcontext));
+	ast_sorcery_object_field_register(sorcery, "global", "contact_expiration_check_interval",
+		__stringify(DEFAULT_CONTACT_EXPIRATION_CHECK_INTERVAL),
+		OPT_UINT_T, 0, FLDSET(struct global_config, contact_expiration_check_interval));
 
 
 	if (ast_sorcery_instance_observer_add(sorcery, &observer_callbacks_global)) {
diff --git a/res/res_pjsip/pjsip_options.c b/res/res_pjsip/pjsip_options.c
index 4cce55836..4713dbb0d 100644
--- a/res/res_pjsip/pjsip_options.c
+++ b/res/res_pjsip/pjsip_options.c
@@ -1078,31 +1078,13 @@ static int qualify_and_schedule_cb(void *obj, void *arg, int flags)
  */
 static int qualify_and_schedule_all_cb(void *obj, void *arg, int flags)
 {
-	struct ast_sip_endpoint *endpoint = obj;
-	char *aors;
-	char *aor_name;
-
-	if (ast_strlen_zero(endpoint->aors)) {
-		return 0;
-	}
-
-	aors = ast_strdupa(endpoint->aors);
-	while ((aor_name = ast_strip(strsep(&aors, ",")))) {
-		struct ast_sip_aor *aor;
-		struct ao2_container *contacts;
-
-		aor = ast_sip_location_retrieve_aor(aor_name);
-		if (!aor) {
-			continue;
-		}
-
-		contacts = ast_sip_location_retrieve_aor_contacts(aor);
-		if (contacts) {
-			ao2_callback(contacts, OBJ_NODATA, qualify_and_schedule_cb, aor);
-			ao2_ref(contacts, -1);
-		}
+	struct ast_sip_aor *aor = obj;
+	struct ao2_container *contacts;
 
-		ao2_ref(aor, -1);
+	contacts = ast_sip_location_retrieve_aor_contacts(aor);
+	if (contacts) {
+		ao2_callback(contacts, OBJ_NODATA, qualify_and_schedule_cb, aor);
+		ao2_ref(contacts, -1);
 	}
 
 	return 0;
@@ -1123,16 +1105,25 @@ static int unschedule_all_cb(void *obj, void *arg, int flags)
 
 static void qualify_and_schedule_all(void)
 {
-	struct ao2_container *endpoints = ast_sip_get_endpoints();
+	struct ast_variable *var = ast_variable_new("qualify_frequency >", "0", "");
+	struct ao2_container *aors;
+
+	if (!var) {
+		return;
+	}
+	aors = ast_sorcery_retrieve_by_fields(ast_sip_get_sorcery(),
+		"aor", AST_RETRIEVE_FLAG_MULTIPLE, var);
+
+	ast_variables_destroy(var);
 
 	ao2_callback(sched_qualifies, OBJ_NODATA | OBJ_MULTIPLE | OBJ_UNLINK, unschedule_all_cb, NULL);
 
-	if (!endpoints) {
+	if (!aors) {
 		return;
 	}
 
-	ao2_callback(endpoints, OBJ_NODATA, qualify_and_schedule_all_cb, NULL);
-	ao2_ref(endpoints, -1);
+	ao2_callback(aors, OBJ_NODATA, qualify_and_schedule_all_cb, NULL);
+	ao2_ref(aors, -1);
 }
 
 static int format_contact_status(void *obj, void *arg, int flags)
diff --git a/res/res_pjsip_mwi.c b/res/res_pjsip_mwi.c
index c9d1b743e..be38b44c1 100644
--- a/res/res_pjsip_mwi.c
+++ b/res/res_pjsip_mwi.c
@@ -966,9 +966,14 @@ static int unsubscribe(void *obj, void *arg, int flags)
 static void create_mwi_subscriptions(void)
 {
 	struct ao2_container *endpoints;
+	struct ast_variable *var;
+
+	var = ast_variable_new("mailboxes !=", "", "");
 
 	endpoints = ast_sorcery_retrieve_by_fields(ast_sip_get_sorcery(), "endpoint",
-		AST_RETRIEVE_FLAG_MULTIPLE | AST_RETRIEVE_FLAG_ALL, NULL);
+		AST_RETRIEVE_FLAG_MULTIPLE, var);
+
+	ast_variables_destroy(var);
 	if (!endpoints) {
 		return;
 	}
diff --git a/res/res_pjsip_registrar_expire.c b/res/res_pjsip_registrar_expire.c
index 399e7bd0e..87edf5390 100644
--- a/res/res_pjsip_registrar_expire.c
+++ b/res/res_pjsip_registrar_expire.c
@@ -25,265 +25,102 @@
 #include "asterisk.h"
 
 #include <pjsip.h>
+#include <sys/time.h>
+#include <signal.h>
 
 #include "asterisk/res_pjsip.h"
 #include "asterisk/module.h"
-#include "asterisk/sched.h"
 
-#define CONTACT_AUTOEXPIRE_BUCKETS 977
+/*! \brief Thread keeping things alive */
+static pthread_t check_thread = AST_PTHREADT_NULL;
 
-static struct ao2_container *contact_autoexpire;
+/*! \brief The global interval at which to check for contact expiration */
+static unsigned int check_interval;
 
-/*! \brief Scheduler used for automatically expiring contacts */
-static struct ast_sched_context *sched;
-
-/*! \brief Structure used for contact auto-expiration */
-struct contact_expiration {
-	/*! \brief Contact that is being auto-expired */
-	struct ast_sip_contact *contact;
-
-	/*! \brief Scheduled item for performing expiration */
-	int sched;
-};
-
-/*! \brief Destructor function for contact auto-expiration */
-static void contact_expiration_destroy(void *obj)
-{
-	struct contact_expiration *expiration = obj;
-
-	ao2_cleanup(expiration->contact);
-}
-
-/*! \brief Hashing function for contact auto-expiration */
-static int contact_expiration_hash(const void *obj, const int flags)
+/*! \brief Callback function which deletes a contact */
+static int expire_contact(void *obj, void *arg, int flags)
 {
-	const struct contact_expiration *object;
-	const char *key;
-
-	switch (flags & OBJ_SEARCH_MASK) {
-	case OBJ_SEARCH_KEY:
-		key = obj;
-		break;
-	case OBJ_SEARCH_OBJECT:
-		object = obj;
-		key = ast_sorcery_object_get_id(object->contact);
-		break;
-	default:
-		/* Hash can only work on something with a full key. */
-		ast_assert(0);
-		return 0;
-	}
-	return ast_str_hash(key);
-}
-
-/*! \brief Comparison function for contact auto-expiration */
-static int contact_expiration_cmp(void *obj, void *arg, int flags)
-{
-	const struct contact_expiration *object_left = obj;
-	const struct contact_expiration *object_right = arg;
-	const char *right_key = arg;
-	int cmp;
-
-	switch (flags & OBJ_SEARCH_MASK) {
-	case OBJ_SEARCH_OBJECT:
-		right_key = ast_sorcery_object_get_id(object_right->contact);
-		/* Fall through */
-	case OBJ_SEARCH_KEY:
-		cmp = strcmp(ast_sorcery_object_get_id(object_left->contact), right_key);
-		break;
-	case OBJ_SEARCH_PARTIAL_KEY:
-		/*
-		 * We could also use a partial key struct containing a length
-		 * so strlen() does not get called for every comparison instead.
-		 */
-		cmp = strncmp(ast_sorcery_object_get_id(object_left->contact), right_key,
-			strlen(right_key));
-		break;
-	default:
-		/*
-		 * What arg points to is specific to this traversal callback
-		 * and has no special meaning to astobj2.
-		 */
-		cmp = 0;
-		break;
-	}
-	if (cmp) {
-		return 0;
-	}
-	/*
-	 * At this point the traversal callback is identical to a sorted
-	 * container.
-	 */
-	return CMP_MATCH;
-}
-
-/*! \brief Scheduler function which deletes a contact */
-static int contact_expiration_expire(const void *data)
-{
-	struct contact_expiration *expiration = (void *) data;
+	struct ast_sip_contact *contact = obj;
 
-	expiration->sched = -1;
+	ast_sorcery_delete(ast_sip_get_sorcery(), contact);
 
-	/* This will end up invoking the deleted observer callback, which will perform the unlinking and such */
-	ast_sorcery_delete(ast_sip_get_sorcery(), expiration->contact);
-	ao2_ref(expiration, -1);
 	return 0;
 }
 
-/*! \brief Observer callback for when a contact is created */
-static void contact_expiration_observer_created(const void *object)
+static void *check_expiration_thread(void *data)
 {
-	const struct ast_sip_contact *contact = object;
-	struct contact_expiration *expiration;
-	int expires = MAX(0, ast_tvdiff_ms(contact->expiration_time, ast_tvnow()));
+	struct ao2_container *contacts;
+	struct ast_variable *var;
+	char *time = alloca(64);
 
-	if (ast_tvzero(contact->expiration_time)) {
-		return;
-	}
+	while (check_interval) {
+		sleep(check_interval);
 
-	expiration = ao2_alloc_options(sizeof(*expiration), contact_expiration_destroy,
-		AO2_ALLOC_OPT_LOCK_NOLOCK);
-	if (!expiration) {
-		return;
-	}
+		sprintf(time, "%ld", ast_tvnow().tv_sec);
+		var = ast_variable_new("expiration_time <=", time, "");
 
-	expiration->contact = (struct ast_sip_contact*)contact;
-	ao2_ref(expiration->contact, +1);
+		ast_debug(4, "Woke up at %s  Interval: %d\n", time, check_interval);
 
-	ao2_ref(expiration, +1);
-	if ((expiration->sched = ast_sched_add(sched, expires, contact_expiration_expire, expiration)) < 0) {
-		ao2_ref(expiration, -1);
-		ast_log(LOG_ERROR, "Scheduled expiration for contact '%s' could not be performed, contact may persist past life\n",
-			ast_sorcery_object_get_id(contact));
-	} else {
-		ao2_link(contact_autoexpire, expiration);
-	}
-	ao2_ref(expiration, -1);
-}
-
-/*! \brief Observer callback for when a contact is updated */
-static void contact_expiration_observer_updated(const void *object)
-{
-	const struct ast_sip_contact *contact = object;
-	struct contact_expiration *expiration;
-	int expires = MAX(0, ast_tvdiff_ms(contact->expiration_time, ast_tvnow()));
+		contacts = ast_sorcery_retrieve_by_fields(ast_sip_get_sorcery(), "contact",
+			AST_RETRIEVE_FLAG_MULTIPLE, var);
 
-	expiration = ao2_find(contact_autoexpire, ast_sorcery_object_get_id(contact),
-		OBJ_SEARCH_KEY);
-	if (!expiration) {
-		return;
+		ast_variables_destroy(var);
+		if (contacts) {
+			ast_debug(3, "Expiring %d contacts\n\n", ao2_container_count(contacts));
+			ao2_callback(contacts, OBJ_NODATA, expire_contact, NULL);
+			ao2_ref(contacts, -1);
+		}
 	}
 
-	AST_SCHED_REPLACE_UNREF(expiration->sched, sched, expires, contact_expiration_expire,
-		expiration, ao2_cleanup(expiration), ao2_cleanup(expiration), ao2_ref(expiration, +1));
-	ao2_ref(expiration, -1);
+	return NULL;
 }
 
-/*! \brief Observer callback for when a contact is deleted */
-static void contact_expiration_observer_deleted(const void *object)
+static void expiration_global_loaded(const char *object_type)
 {
-	struct contact_expiration *expiration;
-
-	expiration = ao2_find(contact_autoexpire, ast_sorcery_object_get_id(object),
-		OBJ_SEARCH_KEY | OBJ_UNLINK);
-	if (!expiration) {
-		return;
-	}
-
-	AST_SCHED_DEL_UNREF(sched, expiration->sched, ao2_cleanup(expiration));
-	ao2_ref(expiration, -1);
-}
-
-/*! \brief Observer callbacks for autoexpiring contacts */
-static const struct ast_sorcery_observer contact_expiration_observer = {
-	.created = contact_expiration_observer_created,
-	.updated = contact_expiration_observer_updated,
-	.deleted = contact_expiration_observer_deleted,
-};
-
-/*! \brief Callback function which deletes a contact if it has expired or sets up auto-expiry */
-static int contact_expiration_setup(void *obj, void *arg, int flags)
-{
-	struct ast_sip_contact *contact = obj;
-	int expires = MAX(0, ast_tvdiff_ms(contact->expiration_time, ast_tvnow()));
-
-	if (!expires) {
-		ast_sorcery_delete(ast_sip_get_sorcery(), contact);
+	check_interval = ast_sip_get_contact_expiration_check_interval();
+
+	/* Observer calls are serialized so this is safe without it's own lock */
+	if (check_interval) {
+		if (check_thread == AST_PTHREADT_NULL) {
+			if (ast_pthread_create_background(&check_thread, NULL, check_expiration_thread, NULL)) {
+				ast_log(LOG_ERROR, "Could not create thread for checking contact expiration.\n");
+				return;
+			}
+			ast_debug(3, "Interval = %d, starting thread\n", check_interval);
+		}
 	} else {
-		contact_expiration_observer_created(contact);
-	}
-
-	return 0;
-}
-
-/*! \brief Initialize auto-expiration of any existing contacts */
-static void contact_expiration_initialize_existing(void)
-{
-	struct ao2_container *contacts;
-
-	contacts = ast_sorcery_retrieve_by_fields(ast_sip_get_sorcery(), "contact",
-		AST_RETRIEVE_FLAG_MULTIPLE | AST_RETRIEVE_FLAG_ALL, NULL);
-	if (!contacts) {
-		return;
+		if (check_thread != AST_PTHREADT_NULL) {
+			pthread_kill(check_thread, SIGURG);
+			check_thread = AST_PTHREADT_NULL;
+			ast_debug(3, "Interval = 0, shutting thread down\n");
+		}
 	}
-
-	ao2_callback(contacts, OBJ_NODATA, contact_expiration_setup, NULL);
-	ao2_ref(contacts, -1);
 }
 
-static int unload_observer_delete(void *obj, void *arg, int flags)
-{
-	struct contact_expiration *expiration = obj;
-
-	AST_SCHED_DEL_UNREF(sched, expiration->sched, ao2_cleanup(expiration));
-	return CMP_MATCH;
-}
+/*! \brief Observer which is used to update our interval when the global setting changes */
+static struct ast_sorcery_observer expiration_global_observer = {
+	.loaded = expiration_global_loaded,
+};
 
 static int unload_module(void)
 {
-	ast_sorcery_observer_remove(ast_sip_get_sorcery(), "contact", &contact_expiration_observer);
-	if (sched) {
-		ao2_callback(contact_autoexpire, OBJ_MULTIPLE | OBJ_NODATA | OBJ_UNLINK,
-			unload_observer_delete, NULL);
-		ast_sched_context_destroy(sched);
-		sched = NULL;
+	if (check_thread != AST_PTHREADT_NULL) {
+		pthread_kill(check_thread, SIGURG);
+		check_thread = AST_PTHREADT_NULL;
 	}
-	ao2_cleanup(contact_autoexpire);
-	contact_autoexpire = NULL;
+
+	ast_sorcery_observer_remove(ast_sip_get_sorcery(), "global", &expiration_global_observer);
 
 	return 0;
 }
 
+
 static int load_module(void)
 {
 	CHECK_PJSIP_MODULE_LOADED();
 
-	contact_autoexpire = ao2_container_alloc_options(AO2_ALLOC_OPT_LOCK_NOLOCK,
-		CONTACT_AUTOEXPIRE_BUCKETS, contact_expiration_hash, contact_expiration_cmp);
-	if (!contact_autoexpire) {
-		ast_log(LOG_ERROR, "Could not create container for contact auto-expiration\n");
-		return AST_MODULE_LOAD_FAILURE;
-	}
-
-	if (!(sched = ast_sched_context_create())) {
-		ast_log(LOG_ERROR, "Could not create scheduler for contact auto-expiration\n");
-		unload_module();
-		return AST_MODULE_LOAD_FAILURE;
-	}
-
-	if (ast_sched_start_thread(sched)) {
-		ast_log(LOG_ERROR, "Could not start scheduler thread for contact auto-expiration\n");
-		unload_module();
-		return AST_MODULE_LOAD_FAILURE;
-	}
-
-	contact_expiration_initialize_existing();
-
-	if (ast_sorcery_observer_add(ast_sip_get_sorcery(), "contact", &contact_expiration_observer)) {
-		ast_log(LOG_ERROR, "Could not add observer for notifications about contacts for contact auto-expiration\n");
-		unload_module();
-		return AST_MODULE_LOAD_FAILURE;
-	}
+	ast_sorcery_observer_add(ast_sip_get_sorcery(), "global", &expiration_global_observer);
+	ast_sorcery_reload_object(ast_sip_get_sorcery(), "global");
 
 	return AST_MODULE_LOAD_SUCCESS;
 }
diff --git a/res/res_sorcery_astdb.c b/res/res_sorcery_astdb.c
index 4e2c3a809..b3642d81e 100644
--- a/res/res_sorcery_astdb.c
+++ b/res/res_sorcery_astdb.c
@@ -63,65 +63,6 @@ static struct ast_sorcery_wizard astdb_object_wizard = {
 	.close = sorcery_astdb_close,
 };
 
-/*! \brief Helper function which converts from a sorcery object set to a json object */
-static struct ast_json *sorcery_objectset_to_json(const struct ast_variable *objectset)
-{
-	struct ast_json *json = ast_json_object_create();
-	const struct ast_variable *field;
-
-	for (field = objectset; field; field = field->next) {
-		struct ast_json *value = ast_json_string_create(field->value);
-
-		if (!value) {
-			ast_json_unref(json);
-			return NULL;
-		} else if (ast_json_object_set(json, field->name, value)) {
-			ast_json_unref(json);
-			return NULL;
-		}
-	}
-
-	return json;
-}
-
-/*! \brief Helper function which converts a json object to a sorcery object set */
-static struct ast_variable *sorcery_json_to_objectset(struct ast_json *json)
-{
-	struct ast_json_iter *field;
-	struct ast_variable *objset = NULL;
-
-	for (field = ast_json_object_iter(json); field; field = ast_json_object_iter_next(json, field)) {
-		struct ast_json *value = ast_json_object_iter_value(field);
-		struct ast_variable *variable = ast_variable_new(ast_json_object_iter_key(field), ast_json_string_get(value), "");
-
-		if (!variable) {
-			ast_variables_destroy(objset);
-			return NULL;
-		}
-
-		variable->next = objset;
-		objset = variable;
-	}
-
-	return objset;
-}
-
-/*! \brief Helper function which compares two json objects and sees if they are equal, but only looks at the criteria provided */
-static int sorcery_json_equal(struct ast_json *object, struct ast_json *criteria)
-{
-	struct ast_json_iter *field;
-
-	for (field = ast_json_object_iter(criteria); field; field = ast_json_object_iter_next(criteria, field)) {
-		struct ast_json *object_field = ast_json_object_get(object, ast_json_object_iter_key(field));
-
-		if (!object_field || !ast_json_equal(object_field, ast_json_object_iter_value(field))) {
-			return 0;
-		}
-	}
-
-	return 1;
-}
-
 static int sorcery_astdb_create(const struct ast_sorcery *sorcery, void *data, void *object)
 {
 	RAII_VAR(struct ast_json *, objset, ast_sorcery_objectset_json_create(sorcery, object), ast_json_unref);
@@ -144,12 +85,11 @@ static void *sorcery_astdb_retrieve_fields_common(const struct ast_sorcery *sorc
 	const char *prefix = data;
 	char family[strlen(prefix) + strlen(type) + 2];
 	RAII_VAR(struct ast_db_entry *, entries, NULL, ast_db_freetree);
-	RAII_VAR(struct ast_json *, criteria, NULL, ast_json_unref);
 	struct ast_db_entry *entry;
 
 	snprintf(family, sizeof(family), "%s/%s", prefix, type);
 
-	if (!(entries = ast_db_gettree(family, NULL)) || (fields && !(criteria = sorcery_objectset_to_json(fields)))) {
+	if (!(entries = ast_db_gettree(family, NULL))) {
 		return NULL;
 	}
 
@@ -158,14 +98,21 @@ static void *sorcery_astdb_retrieve_fields_common(const struct ast_sorcery *sorc
 		RAII_VAR(struct ast_json *, json, NULL, ast_json_unref);
 		struct ast_json_error error;
 		RAII_VAR(struct ast_variable *, objset, NULL, ast_variables_destroy);
+		RAII_VAR(struct ast_variable *, existing, NULL, ast_variables_destroy);
 		void *object = NULL;
 
 		if (!(json = ast_json_load_string(entry->data, &error))) {
 			return NULL;
-		} else if (criteria && !sorcery_json_equal(json, criteria)) {
+		}
+		if (ast_json_to_ast_variables(json, &existing) != AST_JSON_TO_AST_VARS_CODE_SUCCESS) {
+			return NULL;
+		}
+
+		if (fields && !ast_variable_lists_match(existing, fields, 0)) {
 			continue;
-		} else if (!(objset = sorcery_json_to_objectset(json)) ||
-			!(object = ast_sorcery_alloc(sorcery, type, key)) ||
+		}
+
+		if (!(object = ast_sorcery_alloc(sorcery, type, key)) ||
 			ast_sorcery_objectset_apply(sorcery, object, objset)) {
 			ao2_cleanup(object);
 			return NULL;
@@ -199,9 +146,11 @@ static void *sorcery_astdb_retrieve_id(const struct ast_sorcery *sorcery, void *
 
 	snprintf(family, sizeof(family), "%s/%s", prefix, type);
 
-	if (ast_db_get_allocated(family, id, &value) || !(json = ast_json_load_string(value, &error)) ||
-		!(objset = sorcery_json_to_objectset(json)) || !(object = ast_sorcery_alloc(sorcery, type, id)) ||
-		ast_sorcery_objectset_apply(sorcery, object, objset)) {
+	if (ast_db_get_allocated(family, id, &value)
+		|| !(json = ast_json_load_string(value, &error))
+		|| (ast_json_to_ast_variables(json, &objset) != AST_JSON_TO_AST_VARS_CODE_SUCCESS)
+		|| !(object = ast_sorcery_alloc(sorcery, type, id))
+		|| ast_sorcery_objectset_apply(sorcery, object, objset)) {
 		ast_debug(3, "Failed to retrieve object '%s' from astdb\n", id);
 		ao2_cleanup(object);
 		return NULL;
@@ -310,10 +259,10 @@ static void sorcery_astdb_retrieve_regex(const struct ast_sorcery *sorcery, void
 
 		if (regexec(&expression, key, 0, NULL, 0)) {
 			continue;
-		} else if (!(json = ast_json_load_string(entry->data, &error)) ||
-			!(objset = sorcery_json_to_objectset(json)) ||
-			!(object = ast_sorcery_alloc(sorcery, type, key)) ||
-			ast_sorcery_objectset_apply(sorcery, object, objset)) {
+		} else if (!(json = ast_json_load_string(entry->data, &error))
+			|| (ast_json_to_ast_variables(json, &objset) != AST_JSON_TO_AST_VARS_CODE_SUCCESS)
+			|| !(object = ast_sorcery_alloc(sorcery, type, key))
+			|| ast_sorcery_objectset_apply(sorcery, object, objset)) {
 			regfree(&expression);
 			return;
 		}
diff --git a/res/res_sorcery_config.c b/res/res_sorcery_config.c
index 092cc41c8..dd4ea8886 100644
--- a/res/res_sorcery_config.c
+++ b/res/res_sorcery_config.c
@@ -129,7 +129,6 @@ static int sorcery_config_fields_cmp(void *obj, void *arg, int flags)
 {
 	const struct sorcery_config_fields_cmp_params *params = arg;
 	RAII_VAR(struct ast_variable *, objset, NULL, ast_variables_destroy);
-	RAII_VAR(struct ast_variable *, diff, NULL, ast_variables_destroy);
 
 	if (params->regex) {
 		/* If a regular expression has been provided see if it matches, otherwise move on */
@@ -139,11 +138,10 @@ static int sorcery_config_fields_cmp(void *obj, void *arg, int flags)
 		return 0;
 	} else if (params->fields &&
 	    (!(objset = ast_sorcery_objectset_create(params->sorcery, obj)) ||
-	     (ast_sorcery_changeset_create(objset, params->fields, &diff)) ||
-	     diff)) {
+	     (!ast_variable_lists_match(objset, params->fields, 0)))) {
 		/* If we can't turn the object into an object set OR if differences exist between the fields
-	     * passed in and what are present on the object they are not a match.
-	     */
+		 * passed in and what are present on the object they are not a match.
+		 */
 		return 0;
 	}
 
@@ -197,6 +195,7 @@ static void sorcery_config_retrieve_multiple(const struct ast_sorcery *sorcery,
 	if (!config_objects) {
 		return;
 	}
+
 	ao2_callback(config_objects, 0, sorcery_config_fields_cmp, &params);
 }
 
diff --git a/res/res_sorcery_memory.c b/res/res_sorcery_memory.c
index 95cb24835..db1fc1ab8 100644
--- a/res/res_sorcery_memory.c
+++ b/res/res_sorcery_memory.c
@@ -120,7 +120,6 @@ static int sorcery_memory_fields_cmp(void *obj, void *arg, int flags)
 {
 	const struct sorcery_memory_fields_cmp_params *params = arg;
 	RAII_VAR(struct ast_variable *, objset, NULL, ast_variables_destroy);
-	RAII_VAR(struct ast_variable *, diff, NULL, ast_variables_destroy);
 
 	if (params->regex) {
 		/* If a regular expression has been provided see if it matches, otherwise move on */
@@ -130,8 +129,7 @@ static int sorcery_memory_fields_cmp(void *obj, void *arg, int flags)
 		return 0;
 	} else if (params->fields &&
 	    (!(objset = ast_sorcery_objectset_create(params->sorcery, obj)) ||
-	     (ast_sorcery_changeset_create(objset, params->fields, &diff)) ||
-	     diff)) {
+	     (!ast_variable_lists_match(objset, params->fields, 0)))) {
 		/* If we can't turn the object into an object set OR if differences exist between the fields
 		 * passed in and what are present on the object they are not a match.
 		 */
diff --git a/res/res_sorcery_memory_cache.c b/res/res_sorcery_memory_cache.c
index 704372e12..f1fb3c38c 100644
--- a/res/res_sorcery_memory_cache.c
+++ b/res/res_sorcery_memory_cache.c
@@ -1253,8 +1253,7 @@ static int sorcery_memory_cache_fields_cmp(void *obj, void *arg, int flags)
 		}
 		return 0;
 	} else if (params->fields &&
-	     (ast_sorcery_changeset_create(cached->objectset, params->fields, &diff) ||
-	     diff)) {
+	     (!ast_variable_lists_match(cached->objectset, params->fields, 0))) {
 		/* If we can't turn the object into an object set OR if differences exist between the fields
 		 * passed in and what are present on the object they are not a match.
 		 */
diff --git a/res/res_sorcery_realtime.c b/res/res_sorcery_realtime.c
index 83736a102..abf2840fb 100644
--- a/res/res_sorcery_realtime.c
+++ b/res/res_sorcery_realtime.c
@@ -40,6 +40,18 @@ ASTERISK_REGISTER_FILE()
 /*! \brief They key field used to store the unique identifier for the object */
 #define UUID_FIELD "id"
 
+enum unqualified_fetch {
+	UNQUALIFIED_FETCH_NO,
+	UNQUALIFIED_FETCH_WARN,
+	UNQUALIFIED_FETCH_YES,
+	UNQUALIFIED_FETCH_ERROR,
+};
+
+struct sorcery_config {
+	enum unqualified_fetch fetch;
+	char family[];
+};
+
 static void *sorcery_realtime_open(const char *data);
 static int sorcery_realtime_create(const struct ast_sorcery *sorcery, void *data, void *object);
 static void *sorcery_realtime_retrieve_id(const struct ast_sorcery *sorcery, void *data, const char *type, const char *id);
@@ -66,7 +78,7 @@ static struct ast_sorcery_wizard realtime_object_wizard = {
 
 static int sorcery_realtime_create(const struct ast_sorcery *sorcery, void *data, void *object)
 {
-	const char *family = data;
+	struct sorcery_config *config = data;
 	RAII_VAR(struct ast_variable *, fields, ast_sorcery_objectset_create(sorcery, object), ast_variables_destroy);
 	struct ast_variable *id = ast_variable_new(UUID_FIELD, ast_sorcery_object_get_id(object), "");
 
@@ -79,7 +91,7 @@ static int sorcery_realtime_create(const struct ast_sorcery *sorcery, void *data
 	id->next = fields;
 	fields = id;
 
-	return (ast_store_realtime_fields(family, fields) <= 0) ? -1 : 0;
+	return (ast_store_realtime_fields(config->family, fields) <= 0) ? -1 : 0;
 }
 
 /*! \brief Internal helper function which returns a filtered objectset. 
@@ -149,12 +161,12 @@ static struct ast_variable *sorcery_realtime_filter_objectset(struct ast_variabl
 
 static void *sorcery_realtime_retrieve_fields(const struct ast_sorcery *sorcery, void *data, const char *type, const struct ast_variable *fields)
 {
-	const char *family = data;
+	struct sorcery_config *config = data;
 	RAII_VAR(struct ast_variable *, objectset, NULL, ast_variables_destroy);
 	RAII_VAR(struct ast_variable *, id, NULL, ast_variables_destroy);
 	void *object = NULL;
 
-	if (!(objectset = ast_load_realtime_fields(family, fields))) {
+	if (!(objectset = ast_load_realtime_fields(config->family, fields))) {
 		return NULL;
 	}
 
@@ -178,7 +190,7 @@ static void *sorcery_realtime_retrieve_id(const struct ast_sorcery *sorcery, voi
 
 static void sorcery_realtime_retrieve_multiple(const struct ast_sorcery *sorcery, void *data, const char *type, struct ao2_container *objects, const struct ast_variable *fields)
 {
-	const char *family = data;
+	struct sorcery_config *config = data;
 	RAII_VAR(struct ast_config *, rows, NULL, ast_config_destroy);
 	RAII_VAR(struct ast_variable *, all, NULL, ast_variables_destroy);
 	struct ast_category *row = NULL;
@@ -186,6 +198,18 @@ static void sorcery_realtime_retrieve_multiple(const struct ast_sorcery *sorcery
 	if (!fields) {
 		char field[strlen(UUID_FIELD) + 6], value[2];
 
+		if (config->fetch == UNQUALIFIED_FETCH_NO) {
+			return;
+		}
+		if (config->fetch == UNQUALIFIED_FETCH_ERROR) {
+			ast_log(LOG_ERROR, "Unqualified fetch prevented on %s\n", config->family);
+			return;
+		}
+		if (config->fetch == UNQUALIFIED_FETCH_WARN) {
+			ast_log(LOG_WARNING, "Unqualified fetch attempted on %s\n", config->family);
+			return;
+		}
+
 		/* If no fields have been specified we want all rows, so trick realtime into doing it */
 		snprintf(field, sizeof(field), "%s LIKE", UUID_FIELD);
 		snprintf(value, sizeof(value), "%%");
@@ -197,7 +221,7 @@ static void sorcery_realtime_retrieve_multiple(const struct ast_sorcery *sorcery
 		fields = all;
 	}
 
-	if (!(rows = ast_load_realtime_multientry_fields(family, fields))) {
+	if (!(rows = ast_load_realtime_multientry_fields(config->family, fields))) {
 		return;
 	}
 
@@ -221,16 +245,18 @@ static void sorcery_realtime_retrieve_regex(const struct ast_sorcery *sorcery, v
 	char field[strlen(UUID_FIELD) + 6], value[strlen(regex) + 3];
 	RAII_VAR(struct ast_variable *, fields, NULL, ast_variables_destroy);
 
-	/* The realtime API provides no direct ability to do regex so for now we support a limited subset using pattern matching */
-	snprintf(field, sizeof(field), "%s LIKE", UUID_FIELD);
-	if (regex[0] == '^') {
-		snprintf(value, sizeof(value), "%s%%", regex + 1);
-	} else {
-		snprintf(value, sizeof(value), "%%%s%%", regex);
-	}
+	if (!ast_strlen_zero(regex)) {
+		/* The realtime API provides no direct ability to do regex so for now we support a limited subset using pattern matching */
+		snprintf(field, sizeof(field), "%s LIKE", UUID_FIELD);
+		if (regex[0] == '^') {
+			snprintf(value, sizeof(value), "%s%%", regex + 1);
+		} else {
+			snprintf(value, sizeof(value), "%%%s%%", regex);
+		}
 
-	if (!(fields = ast_variable_new(field, value, ""))) {
-		return;
+		if (!(fields = ast_variable_new(field, value, ""))) {
+			return;
+		}
 	}
 
 	sorcery_realtime_retrieve_multiple(sorcery, data, type, objects, fields);
@@ -238,31 +264,74 @@ static void sorcery_realtime_retrieve_regex(const struct ast_sorcery *sorcery, v
 
 static int sorcery_realtime_update(const struct ast_sorcery *sorcery, void *data, void *object)
 {
-	const char *family = data;
+	struct sorcery_config *config = data;
 	RAII_VAR(struct ast_variable *, fields, ast_sorcery_objectset_create(sorcery, object), ast_variables_destroy);
 
 	if (!fields) {
 		return -1;
 	}
 
-	return (ast_update_realtime_fields(family, UUID_FIELD, ast_sorcery_object_get_id(object), fields) <= 0) ? -1 : 0;
+	return (ast_update_realtime_fields(config->family, UUID_FIELD, ast_sorcery_object_get_id(object), fields) <= 0) ? -1 : 0;
 }
 
 static int sorcery_realtime_delete(const struct ast_sorcery *sorcery, void *data, void *object)
 {
-	const char *family = data;
+	struct sorcery_config *config = data;
 
-	return (ast_destroy_realtime_fields(family, UUID_FIELD, ast_sorcery_object_get_id(object), NULL) <= 0) ? -1 : 0;
+	return (ast_destroy_realtime_fields(config->family, UUID_FIELD, ast_sorcery_object_get_id(object), NULL) <= 0) ? -1 : 0;
 }
 
 static void *sorcery_realtime_open(const char *data)
 {
+	struct sorcery_config *config;
+	char *tmp;
+	char *family;
+	char *option;
+
 	/* We require a prefix for family string generation, or else stuff could mix together */
-	if (ast_strlen_zero(data) || !ast_realtime_is_mapping_defined(data)) {
+	if (ast_strlen_zero(data)) {
+		return NULL;
+	}
+
+	tmp = ast_strdupa(data);
+	family = strsep(&tmp, ",");
+
+	if (!ast_realtime_is_mapping_defined(family)) {
+		return NULL;
+	}
+
+	config = ast_calloc(1, sizeof(*config) + strlen(family) + 1);
+	if (!config) {
 		return NULL;
 	}
 
-	return ast_strdup(data);
+	strcpy(config->family, family); /* Safe */
+	config->fetch = UNQUALIFIED_FETCH_YES;
+
+	while ((option = strsep(&tmp, ","))) {
+		char *name = strsep(&option, "=");
+		char *value = option;
+
+		if (!strcasecmp(name, "allow_unqualified_fetch")) {
+			if (ast_strlen_zero(value) || !strcasecmp(value, "yes")) {
+				config->fetch = UNQUALIFIED_FETCH_YES;
+			} else if (!strcasecmp(value, "no")) {
+				config->fetch = UNQUALIFIED_FETCH_NO;
+			} else if (!strcasecmp(value, "warn")) {
+				config->fetch = UNQUALIFIED_FETCH_WARN;
+			} else if (!strcasecmp(value, "error")) {
+				config->fetch = UNQUALIFIED_FETCH_ERROR;
+			} else {
+				ast_log(LOG_ERROR, "Unrecognized value in %s:%s: '%s'\n", family, name, value);
+				return NULL;
+			}
+		} else {
+			ast_log(LOG_ERROR, "Unrecognized option in %s: '%s'\n", family, name);
+			return NULL;
+		}
+	}
+
+	return config;
 }
 
 static void sorcery_realtime_close(void *data)