sorcery/res_pjsip: Refactor for realtime performance

There were a number of places in the res_pjsip stack that were getting
all endpoints or all aors, and then filtering them locally.

A good example is pjsip_options which, on startup, retrieves all
endpoints, then the aors for those endpoints, then tests the aors to see
if the qualify_frequency is > 0.  One issue was that it never did
anything with the endpoints other than retrieve the aors so we probably
could have skipped a step and just retrieved all aors. But nevermind.

This worked reasonably well with local config files but with a realtime
backend and thousands of objects, this was a nightmare.  The issue
really boiled down to the fact that while realtime supports predicates
that are passed to the database engine, the non-realtime sorcery
backends didn't.

They do now.

The realtime engines have a scheme for doing simple comparisons. They
take in an ast_variable (or list) for matching, and the name of each
variable can contain an operator.  For instance, a name of
"qualify_frequency >" and a value of "0" would create a SQL predicate
that looks like "where qualify_frequency > '0'".  If there's no operator
after the name, the engines add an '=' so a simple name of
"qualify_frequency" and a value of "10" would return exact matches.

The non-realtime backends decide whether to include an object in a
result set by calling ast_sorcery_changeset_create on every object in
the internal container.  However, ast_sorcery_changeset_create only does
exact string matches though so a name of "qualify_frequency >" and a
value of "0" returns nothing because the literal "qualify_frequency >"
doesn't match any name in the objset set.

So, the real task was to create a generic string matcher that can take a
left value, operator and a right value and perform the match. To that
end, strings.c has a new ast_strings_match(left, operator, right)
function.  Left and right are the strings to operate on and the operator
can be a string containing any of the following: = (or NULL or ""), !=,
>, >=, <, <=, like or regex.  If the operator is like or regex, the
right string should be a %-pattern or a regex expression.  If both left
and right can be converted to float, then a numeric comparison is
performed, otherwise a string comparison is performed.

To use this new function on ast_variables, 2 new functions were added to
config.c.  One that compares 2 ast_variables, and one that compares 2
ast_variable lists.  The former is useful when you want to compare 2
ast_variables that happen to be in a list but don't want to traverse the
list.  The latter will traverse the right list and return true if all
the variables in it match the left list.

Now, the backends' fields_cmp functions call ast_variable_lists_match
instead of ast_sorcery_changeset_create and they can now process the
same syntax as the realtime engines.  The realtime backend just passes
the variable list unaltered to the engine.  The only gotcha is that
there's no common realtime engine support for regex so that's been noted
in the api docs for ast_sorcery_retrieve_by_fields.

Only one more change to sorcery was done...  A new config flag
"allow_unqualified_fetch" was added to reg_sorcery_realtime.
"no": ignore fetches if no predicate fields were supplied.
"error": same as no but emit an error. (good for testing)
"yes": allow (the default);
"warn": allow but emit a warning. (good for testing)

Now on to res_pjsip...

pjsip_options was modified to retrieve aors with qualify_frequency > 0
rather than all endpoints then all aors.  Not only was this a big
improvement in realtime retrieval but even for config files there's an
improvement because we're not going through endpoints anymore.

res_pjsip_mwi was modified to retieve only endpoints with something in
the mailboxes field instead of all endpoints then testing mailboxes.

res_pjsip_registrar_expire was completely refactored.  It was retrieving
all contacts then setting up scheduler entries to check for expiration.
Now, it's a single thread (like keepalive) that periodically retrieves
only contacts whose expiration time is < now and deletes them.  A new
contact_expiration_check_interval was added to global with a default of
30 seconds.

Ross Beer reports that with this patch, his Asterisk startup time dropped
from around an hour to under 30 seconds.

There are still objects that can't be filtered at the database like
identifies, transports, and registrations.  These are not going to be
anywhere near as numerous as endpoints, aors, auths, contacts however.

Back to allow_unqualified_fetch.  If this is set to yes and you have a
very large number of objects in the database, the pjsip CLI commands
will attempt to retrive ALL of them if not qualified with a LIKE.
Worse, if you type "pjsip show endpoint <tab>" guess what's going to
happen? :)  Having a cache helps but all the objects will have to be
retrieved at least once to fill the cache.  Setting
allow_unqualified_fetch=no prevents the mass retrieve and should be used
on endpoints, auths, aors, and contacts.  It should NOT be used for
identifies, registrations and transports since these MUST be
retrieved in bulk.

Example sorcery.conf:

[res_pjsip]
endpoint=config,pjsip.conf,criteria=type=endpoint
endpoint=realtime,ps_endpoints,allow_unqualified_fetch=error

ASTERISK-25826 #close
Reported-by: Ross Beer
Tested-by: Ross Beer

Change-Id: Id2691e447db90892890036e663aaf907b2dc1c67

22 files changed, 877 insertions, 421 deletions

diff --git a/configs/samples/pjsip.conf.sample b/configs/samples/pjsip.conf.sample
index ebbd199e7..c7b33c025 100644
--- a/configs/samples/pjsip.conf.sample
+++ b/configs/samples/pjsip.conf.sample
@@ -886,6 +886,8 @@
 ;keep_alive_interval=20 ; The interval (in seconds) at which to send keepalive
                         ; messages on all active connection-oriented transports
                         ; (default: "0")
+;contact_expiration_check_interval=30
+                        ; The interval (in seconds) to check for expired contacts.
 ;endpoint_identifier_order=ip,username,anonymous
             ; The order by which endpoint identifiers are given priority.
             ; Identifier names are derived from res_pjsip_endpoint_identifier_*
diff --git a/contrib/ast-db-manage/config/versions/5813202e92be_add_contact_expiration_check_interval_.py b/contrib/ast-db-manage/config/versions/5813202e92be_add_contact_expiration_check_interval_.py
new file mode 100644
index 000000000..2c61f2b9d
--- /dev/null
+++ b/contrib/ast-db-manage/config/versions/5813202e92be_add_contact_expiration_check_interval_.py
@@ -0,0 +1,21 @@
+"""Add contact_expiration_check_interval to ps_globals
+
+Revision ID: 5813202e92be
+Revises: 3bcc0b5bc2c9
+Create Date: 2016-03-08 21:52:21.372310
+
+"""
+
+# revision identifiers, used by Alembic.
+revision = '5813202e92be'
+down_revision = '3bcc0b5bc2c9'
+
+from alembic import op
+import sqlalchemy as sa
+
+def upgrade():
+    op.add_column('ps_globals', sa.Column('contact_expiration_check_interval', sa.Integer))
+
+def downgrade():
+    with op.batch_alter_table('ps_globals') as batch_op:
+        batch_op.drop_column('contact_expiration_check_interval')
diff --git a/include/asterisk/config.h b/include/asterisk/config.h
index d0bcae654..4c68e5c21 100644
--- a/include/asterisk/config.h
+++ b/include/asterisk/config.h
@@ -307,7 +307,7 @@ const char *ast_variable_retrieve(struct ast_config *config,
 	const char *category, const char *variable);
 
 /*!
- * \brief Gets a variable from a specific category structure
+ * \brief Gets a variable value from a specific category structure by name
  *
  * \param category category structure under which the variable lies
  * \param variable which variable you wish to get the data for
@@ -321,7 +321,7 @@ const char *ast_variable_retrieve(struct ast_config *config,
 const char *ast_variable_find(const struct ast_category *category, const char *variable);
 
 /*!
- * \brief Gets a variable from a variable list
+ * \brief Gets the value of a variable from a variable list by name
  *
  * \param list variable list to search
  * \param variable which variable you wish to get the data for
@@ -335,7 +335,7 @@ const char *ast_variable_find(const struct ast_category *category, const char *v
 const char *ast_variable_find_in_list(const struct ast_variable *list, const char *variable);
 
 /*!
- * \brief Gets the LAST occurrence of a variable from a variable list
+ * \brief Gets the value of the LAST occurrence of a variable from a variable list
  *
  * \param list The ast_variable list to search
  * \param variable The name of the ast_variable you wish to fetch data for
@@ -352,6 +352,21 @@ const char *ast_variable_find_in_list(const struct ast_variable *list, const cha
 const char *ast_variable_find_last_in_list(const struct ast_variable *list, const char *variable);
 
 /*!
+ * \brief Gets a variable from a variable list by name
+ * \since 13.9.0
+ *
+ * \param list variable list to search
+ * \param variable name you wish to get the data for
+ *
+ * \details
+ * Goes through a given variable list and searches for the given variable
+ *
+ * \retval The variable (not the value) on success
+ * \retval NULL if unable to find it.
+ */
+const struct ast_variable *ast_variable_find_variable_in_list(const struct ast_variable *list, const char *variable_name);
+
+/*!
  * \brief Retrieve a category if it exists
  *
  * \param config which config to use
@@ -1217,6 +1232,66 @@ char *ast_realtime_decode_chunk(char *chunk);
  */
 char *ast_realtime_encode_chunk(struct ast_str **dest, ssize_t maxlen, const char *chunk);
 
+/*!
+ * \brief Tests 2 variable values to see if they match
+ * \since 13.9.0
+ *
+ * \param left Variable to test
+ * \param right Variable to match against with an optional realtime-style operator in the name
+ *
+ * \retval 1 matches
+ * \retval 0 doesn't match
+ *
+ * \details
+ *
+ * The values of the variables are passed to ast_strings_match.
+ * If right->name is suffixed with a space and an operator, that operator
+ * is also passed to ast_strings_match.
+ *
+ * Examples:
+ *
+ * left->name = "id" (ignored)
+ * left->value = "abc"
+ * right->name = "id regex" (id is ignored)
+ * right->value = "a[bdef]c"
+ *
+ * will result in ast_strings_match("abc", "regex", "a[bdef]c") which will return 1.
+ *
+ * left->name = "id" (ignored)
+ * left->value = "abc"
+ * right->name = "id" (ignored)
+ * right->value = "abc"
+ *
+ * will result in ast_strings_match("abc", NULL, "abc") which will return 1.
+ *
+ * See the documentation for ast_strings_match for the valid operators.
+ */
+int ast_variables_match(const struct ast_variable *left, const struct ast_variable *right);
+
+/*!
+ * \brief Tests 2 variable lists to see if they match
+ * \since 13.9.0
+ *
+ * \param left Variable list to test
+ * \param right Variable list with an optional realtime-style operator in the names
+ * \param exact_match If true, all variables in left must match all variables in right
+ *        and vice versa.  This does exact value matches only.  Operators aren't supported.
+ *        Except for order, the left and right lists must be equal.
+ *
+ *        If false, every variable in the right list must match some variable in the left list
+ *        using the operators supplied. Variables in the left list that aren't in the right
+ *        list are ignored for matching purposes.
+ *
+ * \retval 1 matches
+ * \retval 0 doesn't match
+ *
+ * \details
+ * Iterates over the variable lists calling ast_variables_match.  If any match fails
+ * or a variable in the right list isn't in the left list, 0 is returned.
+ */
+int ast_variable_lists_match(const struct ast_variable *left, const struct ast_variable *right,
+	int exact_match);
+
 #if defined(__cplusplus) || defined(c_plusplus)
 }
 #endif
diff --git a/include/asterisk/res_pjsip.h b/include/asterisk/res_pjsip.h
index 3901cf73c..50fbc5102 100644
--- a/include/asterisk/res_pjsip.h
+++ b/include/asterisk/res_pjsip.h
@@ -2193,6 +2193,14 @@ void ast_sip_get_default_from_user(char *from_user, size_t size);
 unsigned int ast_sip_get_keep_alive_interval(void);
 
 /*!
+ * \brief Retrieve the system contact expiration check interval setting.
+ *
+ * \retval the contact expiration check interval.
+ */
+unsigned int ast_sip_get_contact_expiration_check_interval(void);
+
+
+/*!
  * \brief Retrieve the system max initial qualify time.
  *
  * \retval the maximum initial qualify time.
diff --git a/include/asterisk/sorcery.h b/include/asterisk/sorcery.h
index 027ec0058..5e947257b 100644
--- a/include/asterisk/sorcery.h
+++ b/include/asterisk/sorcery.h
@@ -1148,6 +1148,7 @@ void *ast_sorcery_retrieve_by_id(const struct ast_sorcery *sorcery, const char *
 
 /*!
  * \brief Retrieve an object or multiple objects using specific fields
+ * \since 13.9.0
  *
  * \param sorcery Pointer to a sorcery structure
  * \param type Type of object to retrieve
@@ -1162,6 +1163,29 @@ void *ast_sorcery_retrieve_by_id(const struct ast_sorcery *sorcery, const char *
  *
  * \note If the AST_RETRIEVE_FLAG_ALL flag is used you may omit fields to retrieve all objects
  *       of the given type.
+ *
+ * \note The fields parameter can contain realtime-style expressions in variable->name.
+ *       All operators defined for ast_strings_match can be used except for regex as
+ *       there's no common support for regex in the realtime backends at this time.
+ *       If multiple variables are in the fields list, all must match for an object to
+ *       be returned.  See ast_strings_match for more information.
+ *
+ * Example:
+ *
+ * The following code can be significantly faster when a realtime backend is in use
+ * because the expression "qualify_frequency > 0" is passed to the database to limit
+ * the number of rows returned.
+ *
+ *  struct ast_variable *var = ast_variable_new("qualify_frequency >", "0", "");
+ *  struct ao2_container *aors;
+ *
+ *  if (!var) {
+ *  	return;
+ *  }
+ *
+ *  aors = ast_sorcery_retrieve_by_fields(ast_sip_get_sorcery(),
+ *      "aor", AST_RETRIEVE_FLAG_MULTIPLE, var);
+ *
  */
 void *ast_sorcery_retrieve_by_fields(const struct ast_sorcery *sorcery, const char *type, unsigned int flags, struct ast_variable *fields);
 
diff --git a/include/asterisk/strings.h b/include/asterisk/strings.h
index 3701b5305..0e2f69ba8 100644
--- a/include/asterisk/strings.h
+++ b/include/asterisk/strings.h
@@ -1335,4 +1335,34 @@ void ast_str_container_remove(struct ao2_container *str_container, const char *r
  * \return A pointer to buf
  */
 char *ast_generate_random_string(char *buf, size_t size);
+
+/*!
+ * \brief Compares 2 strings using realtime-style operators
+ * \since 13.9.0
+ *
+ * \param left The left side of the equation
+ * \param op The operator to apply
+ * \param right The right side of the equation
+ *
+ * \retval 1 matches
+ * \retval 0 doesn't match
+ *
+ * \details
+ *
+ * Operators:
+ * 	"=", "!=", "<", "<=", ">", ">=":
+ * 	   If both left and right can be converted to float, then they will be
+ * 	   compared as such. Otherwise the result will be derived from strcmp(left, right).
+ * "regex":
+ *     The right value will be compiled as a regular expression and matched against the left
+ *     value.
+ * "like":
+ *     Any '%' character in the right value will be converted to '.*' and the resulting
+ *     string will be handled as a regex.
+ * NULL , "":
+ *     If the right value starts and ends with a '/' then it will be processed as a regex.
+ *     Otherwise, same as "=".
+ */
+int ast_strings_match(const char *left, const char *op, const char *right);
+
 #endif /* _ASTERISK_STRINGS_H */
diff --git a/main/config.c b/main/config.c
index 74f42cf72..69621868c 100644
--- a/main/config.c
+++ b/main/config.c
@@ -723,6 +723,96 @@ const char *ast_variable_find(const struct ast_category *category, const char *v
 	return ast_variable_find_in_list(category->root, variable);
 }
 
+const struct ast_variable *ast_variable_find_variable_in_list(const struct ast_variable *list, const char *variable_name)
+{
+	const struct ast_variable *v;
+
+	for (v = list; v; v = v->next) {
+		if (!strcasecmp(variable_name, v->name)) {
+			return v;
+		}
+	}
+	return NULL;
+}
+
+int ast_variables_match(const struct ast_variable *left, const struct ast_variable *right)
+{
+	char *op;
+
+	if (left == right) {
+		return 1;
+	}
+
+	if (!(left && right)) {
+		return 0;
+	}
+
+	op = strrchr(right->name, ' ');
+	if (op) {
+		op++;
+	}
+
+	return ast_strings_match(left->value, op ? ast_strdupa(op) : NULL, right->value);
+}
+
+int ast_variable_lists_match(const struct ast_variable *left, const struct ast_variable *right, int exact_match)
+{
+	const struct ast_variable *field;
+	int right_count = 0;
+	int left_count = 0;
+
+	if (left == right) {
+		return 1;
+	}
+
+	if (!(left && right)) {
+		return 0;
+	}
+
+	for (field = right; field; field = field->next) {
+		char *space = strrchr(field->name, ' ');
+		const struct ast_variable *old;
+		char * name = (char *)field->name;
+
+		if (space) {
+			name = ast_strdup(field->name);
+			if (!name) {
+				return 0;
+			}
+			name[space - field->name] = '\0';
+		}
+
+		old = ast_variable_find_variable_in_list(left, name);
+		if (name != field->name) {
+			ast_free(name);
+		}
+
+		if (exact_match) {
+			if (!old || strcmp(old->value, field->value)) {
+				return 0;
+			}
+		} else {
+			if (!ast_variables_match(old, field)) {
+				return 0;
+			}
+		}
+
+		right_count++;
+	}
+
+	if (exact_match) {
+		for (field = left; field; field = field->next) {
+			left_count++;
+		}
+
+		if (right_count != left_count) {
+			return 0;
+		}
+	}
+
+	return 1;
+}
+
 const char *ast_variable_find_in_list(const struct ast_variable *list, const char *variable)
 {
 	const struct ast_variable *v;
diff --git a/main/strings.c b/main/strings.c
index 53d50954f..9e885ebc3 100644
--- a/main/strings.c
+++ b/main/strings.c
@@ -39,6 +39,7 @@
 
 ASTERISK_FILE_VERSION(__FILE__, "$Revision$")
 
+#include <regex.h>
 #include "asterisk/strings.h"
 #include "asterisk/pbx.h"
 
@@ -228,3 +229,129 @@ char *ast_generate_random_string(char *buf, size_t size)
 
 	return buf;
 }
+
+int ast_strings_match(const char *left, const char *op, const char *right)
+{
+	char *internal_op = (char *)op;
+	char *internal_right = (char *)right;
+	float left_num;
+	float right_num;
+	int scan_numeric = 0;
+
+	if (!(left && right)) {
+		return 0;
+	}
+
+	if (ast_strlen_zero(op)) {
+		if (ast_strlen_zero(left) && ast_strlen_zero(right)) {
+			return 1;
+		}
+
+		if (strlen(right) >= 2 && right[0] == '/' && right[strlen(right) - 1] == '/') {
+			internal_op = "regex";
+			internal_right = ast_strdupa(right);
+			/* strip the leading and trailing '/' */
+			internal_right++;
+			internal_right[strlen(internal_right) - 1] = '\0';
+			goto regex;
+		} else {
+			internal_op = "=";
+			goto equals;
+		}
+	}
+
+	if (!strcasecmp(op, "like")) {
+		char *tok;
+		struct ast_str *buffer = ast_str_alloca(128);
+
+		if (!strchr(right, '%')) {
+			return !strcmp(left, right);
+		} else {
+			internal_op = "regex";
+			internal_right = ast_strdupa(right);
+			tok = strsep(&internal_right, "%");
+			ast_str_set(&buffer, 0, "^%s", tok);
+
+			while ((tok = strsep(&internal_right, "%"))) {
+				ast_str_append(&buffer, 0, ".*%s", tok);
+			}
+			ast_str_append(&buffer, 0, "%s", "$");
+
+			internal_right = ast_str_buffer(buffer);
+			/* fall through to regex */
+		}
+	}
+
+regex:
+	if (!strcasecmp(internal_op, "regex")) {
+		regex_t expression;
+		int rc;
+
+		if (regcomp(&expression, internal_right, REG_EXTENDED | REG_NOSUB)) {
+			return 0;
+		}
+
+		rc = regexec(&expression, left, 0, NULL, 0);
+		regfree(&expression);
+		return !rc;
+	}
+
+equals:
+	scan_numeric = (sscanf(left, "%f", &left_num) && sscanf(internal_right, "%f", &right_num));
+
+	if (internal_op[0] == '=') {
+		if (ast_strlen_zero(left) && ast_strlen_zero(internal_right)) {
+			return 1;
+		}
+
+		if (scan_numeric) {
+			return (left_num == right_num);
+		} else {
+			return (!strcmp(left, internal_right));
+		}
+	}
+
+	if (internal_op[0] == '!' && internal_op[1] == '=') {
+		if (scan_numeric) {
+			return (left_num != right_num);
+		} else {
+			return !!strcmp(left, internal_right);
+		}
+	}
+
+	if (internal_op[0] == '<') {
+		if (scan_numeric) {
+			if (internal_op[1] == '=') {
+				return (left_num <= right_num);
+			} else {
+				return (left_num < right_num);
+			}
+		} else {
+			if (internal_op[1] == '=') {
+				return strcmp(left, internal_right) <= 0;
+			} else {
+				return strcmp(left, internal_right) < 0;
+			}
+		}
+	}
+
+	if (internal_op[0] == '>') {
+		if (scan_numeric) {
+			if (internal_op[1] == '=') {
+				return (left_num >= right_num);
+			} else {
+				return (left_num > right_num);
+			}
+		} else {
+			if (internal_op[1] == '=') {
+				return strcmp(left, internal_right) >= 0;
+			} else {
+				return strcmp(left, internal_right) > 0;
+			}
+		}
+	}
+
+	return 0;
+}
+
+
diff --git a/res/res_pjsip.c b/res/res_pjsip.c
index 752491c1d..e84da1a45 100644
--- a/res/res_pjsip.c
+++ b/res/res_pjsip.c
@@ -1279,6 +1279,9 @@
 				<configOption name="keep_alive_interval" default="0">
 					<synopsis>The interval (in seconds) to send keepalives to active connection-oriented transports.</synopsis>
 				</configOption>
+				<configOption name="contact_expiration_check_interval" default="30">
+					<synopsis>The interval (in seconds) to check for expired contacts.</synopsis>
+				</configOption>
 				<configOption name="max_initial_qualify_time" default="0">
 					<synopsis>The maximum amount of time from startup that qualifies should be attempted on all contacts.
 					If greater than the qualify_frequency for an aor, qualify_frequency will be used instead.</synopsis>
diff --git a/res/res_pjsip/config_global.c b/res/res_pjsip/config_global.c
index 3d88ffc2a..c0fede64d 100644
--- a/res/res_pjsip/config_global.c
+++ b/res/res_pjsip/config_global.c
@@ -36,6 +36,7 @@
 #define DEFAULT_MAX_INITIAL_QUALIFY_TIME 0
 #define DEFAULT_FROM_USER "asterisk"
 #define DEFAULT_REGCONTEXT ""
+#define DEFAULT_CONTACT_EXPIRATION_CHECK_INTERVAL 30
 
 static char default_useragent[256];
 
@@ -58,6 +59,8 @@ struct global_config {
 	unsigned int keep_alive_interval;
 	/* The maximum time for all contacts to be qualified at startup */
 	unsigned int max_initial_qualify_time;
+	/* The interval at which to check for expired contacts */
+	unsigned int contact_expiration_check_interval;
 };
 
 static void global_destructor(void *obj)
@@ -186,6 +189,21 @@ unsigned int ast_sip_get_keep_alive_interval(void)
 	return interval;
 }
 
+unsigned int ast_sip_get_contact_expiration_check_interval(void)
+{
+	unsigned int interval;
+	struct global_config *cfg;
+
+	cfg = get_global_cfg();
+	if (!cfg) {
+		return DEFAULT_CONTACT_EXPIRATION_CHECK_INTERVAL;
+	}
+
+	interval = cfg->contact_expiration_check_interval;
+	ao2_ref(cfg, -1);
+	return interval;
+}
+
 unsigned int ast_sip_get_max_initial_qualify_time(void)
 {
 	unsigned int time;
@@ -331,6 +349,9 @@ int ast_sip_initialize_sorcery_global(void)
 		OPT_STRINGFIELD_T, 0, STRFLDSET(struct global_config, default_from_user));
 	ast_sorcery_object_field_register(sorcery, "global", "regcontext", DEFAULT_REGCONTEXT,
                 OPT_STRINGFIELD_T, 0, STRFLDSET(struct global_config, regcontext));
+	ast_sorcery_object_field_register(sorcery, "global", "contact_expiration_check_interval",
+		__stringify(DEFAULT_CONTACT_EXPIRATION_CHECK_INTERVAL),
+		OPT_UINT_T, 0, FLDSET(struct global_config, contact_expiration_check_interval));
 
 
 	if (ast_sorcery_instance_observer_add(sorcery, &observer_callbacks_global)) {
diff --git a/res/res_pjsip/pjsip_options.c b/res/res_pjsip/pjsip_options.c
index 157a07741..fa0873984 100644
--- a/res/res_pjsip/pjsip_options.c
+++ b/res/res_pjsip/pjsip_options.c
@@ -1089,31 +1089,13 @@ static int qualify_and_schedule_cb(void *obj, void *arg, int flags)
  */
 static int qualify_and_schedule_all_cb(void *obj, void *arg, int flags)
 {
-	struct ast_sip_endpoint *endpoint = obj;
-	char *aors;
-	char *aor_name;
-
-	if (ast_strlen_zero(endpoint->aors)) {
-		return 0;
-	}
-
-	aors = ast_strdupa(endpoint->aors);
-	while ((aor_name = ast_strip(strsep(&aors, ",")))) {
-		struct ast_sip_aor *aor;
-		struct ao2_container *contacts;
-
-		aor = ast_sip_location_retrieve_aor(aor_name);
-		if (!aor) {
-			continue;
-		}
-
-		contacts = ast_sip_location_retrieve_aor_contacts(aor);
-		if (contacts) {
-			ao2_callback(contacts, OBJ_NODATA, qualify_and_schedule_cb, aor);
-			ao2_ref(contacts, -1);
-		}
+	struct ast_sip_aor *aor = obj;
+	struct ao2_container *contacts;
 
-		ao2_ref(aor, -1);
+	contacts = ast_sip_location_retrieve_aor_contacts(aor);
+	if (contacts) {
+		ao2_callback(contacts, OBJ_NODATA, qualify_and_schedule_cb, aor);
+		ao2_ref(contacts, -1);
 	}
 
 	return 0;
@@ -1134,16 +1116,25 @@ static int unschedule_all_cb(void *obj, void *arg, int flags)
 
 static void qualify_and_schedule_all(void)
 {
-	struct ao2_container *endpoints = ast_sip_get_endpoints();
+	struct ast_variable *var = ast_variable_new("qualify_frequency >", "0", "");
+	struct ao2_container *aors;
+
+	if (!var) {
+		return;
+	}
+	aors = ast_sorcery_retrieve_by_fields(ast_sip_get_sorcery(),
+		"aor", AST_RETRIEVE_FLAG_MULTIPLE, var);
+
+	ast_variables_destroy(var);
 
 	ao2_callback(sched_qualifies, OBJ_NODATA | OBJ_MULTIPLE | OBJ_UNLINK, unschedule_all_cb, NULL);
 
-	if (!endpoints) {
+	if (!aors) {
 		return;
 	}
 
-	ao2_callback(endpoints, OBJ_NODATA, qualify_and_schedule_all_cb, NULL);
-	ao2_ref(endpoints, -1);
+	ao2_callback(aors, OBJ_NODATA, qualify_and_schedule_all_cb, NULL);
+	ao2_ref(aors, -1);
 }
 
 static int format_contact_status(void *obj, void *arg, int flags)
diff --git a/res/res_pjsip_mwi.c b/res/res_pjsip_mwi.c
index 7bec75819..b77981cfe 100644
--- a/res/res_pjsip_mwi.c
+++ b/res/res_pjsip_mwi.c
@@ -966,9 +966,14 @@ static int unsubscribe(void *obj, void *arg, int flags)
 static void create_mwi_subscriptions(void)
 {
 	struct ao2_container *endpoints;
+	struct ast_variable *var;
+
+	var = ast_variable_new("mailboxes !=", "", "");
 
 	endpoints = ast_sorcery_retrieve_by_fields(ast_sip_get_sorcery(), "endpoint",
-		AST_RETRIEVE_FLAG_MULTIPLE | AST_RETRIEVE_FLAG_ALL, NULL);
+		AST_RETRIEVE_FLAG_MULTIPLE, var);
+
+	ast_variables_destroy(var);
 	if (!endpoints) {
 		return;
 	}
diff --git a/res/res_pjsip_registrar_expire.c b/res/res_pjsip_registrar_expire.c
index 5b2b350ea..bdb86e50e 100644
--- a/res/res_pjsip_registrar_expire.c
+++ b/res/res_pjsip_registrar_expire.c
@@ -25,265 +25,102 @@
 #include "asterisk.h"
 
 #include <pjsip.h>
+#include <sys/time.h>
+#include <signal.h>
 
 #include "asterisk/res_pjsip.h"
 #include "asterisk/module.h"
-#include "asterisk/sched.h"
 
-#define CONTACT_AUTOEXPIRE_BUCKETS 977
+/*! \brief Thread keeping things alive */
+static pthread_t check_thread = AST_PTHREADT_NULL;
 
-static struct ao2_container *contact_autoexpire;
+/*! \brief The global interval at which to check for contact expiration */
+static unsigned int check_interval;
 
-/*! \brief Scheduler used for automatically expiring contacts */
-static struct ast_sched_context *sched;
-
-/*! \brief Structure used for contact auto-expiration */
-struct contact_expiration {
-	/*! \brief Contact that is being auto-expired */
-	struct ast_sip_contact *contact;
-
-	/*! \brief Scheduled item for performing expiration */
-	int sched;
-};
-
-/*! \brief Destructor function for contact auto-expiration */
-static void contact_expiration_destroy(void *obj)
-{
-	struct contact_expiration *expiration = obj;
-
-	ao2_cleanup(expiration->contact);
-}
-
-/*! \brief Hashing function for contact auto-expiration */
-static int contact_expiration_hash(const void *obj, const int flags)
+/*! \brief Callback function which deletes a contact */
+static int expire_contact(void *obj, void *arg, int flags)
 {
-	const struct contact_expiration *object;
-	const char *key;
-
-	switch (flags & OBJ_SEARCH_MASK) {
-	case OBJ_SEARCH_KEY:
-		key = obj;
-		break;
-	case OBJ_SEARCH_OBJECT:
-		object = obj;
-		key = ast_sorcery_object_get_id(object->contact);
-		break;
-	default:
-		/* Hash can only work on something with a full key. */
-		ast_assert(0);
-		return 0;
-	}
-	return ast_str_hash(key);
-}
-
-/*! \brief Comparison function for contact auto-expiration */
-static int contact_expiration_cmp(void *obj, void *arg, int flags)
-{
-	const struct contact_expiration *object_left = obj;
-	const struct contact_expiration *object_right = arg;
-	const char *right_key = arg;
-	int cmp;
-
-	switch (flags & OBJ_SEARCH_MASK) {
-	case OBJ_SEARCH_OBJECT:
-		right_key = ast_sorcery_object_get_id(object_right->contact);
-		/* Fall through */
-	case OBJ_SEARCH_KEY:
-		cmp = strcmp(ast_sorcery_object_get_id(object_left->contact), right_key);
-		break;
-	case OBJ_SEARCH_PARTIAL_KEY:
-		/*
-		 * We could also use a partial key struct containing a length
-		 * so strlen() does not get called for every comparison instead.
-		 */
-		cmp = strncmp(ast_sorcery_object_get_id(object_left->contact), right_key,
-			strlen(right_key));
-		break;
-	default:
-		/*
-		 * What arg points to is specific to this traversal callback
-		 * and has no special meaning to astobj2.
-		 */
-		cmp = 0;
-		break;
-	}
-	if (cmp) {
-		return 0;
-	}
-	/*
-	 * At this point the traversal callback is identical to a sorted
-	 * container.
-	 */
-	return CMP_MATCH;
-}
-
-/*! \brief Scheduler function which deletes a contact */
-static int contact_expiration_expire(const void *data)
-{
-	struct contact_expiration *expiration = (void *) data;
+	struct ast_sip_contact *contact = obj;
 
-	expiration->sched = -1;
+	ast_sorcery_delete(ast_sip_get_sorcery(), contact);
 
-	/* This will end up invoking the deleted observer callback, which will perform the unlinking and such */
-	ast_sorcery_delete(ast_sip_get_sorcery(), expiration->contact);
-	ao2_ref(expiration, -1);
 	return 0;
 }
 
-/*! \brief Observer callback for when a contact is created */
-static void contact_expiration_observer_created(const void *object)
+static void *check_expiration_thread(void *data)
 {
-	const struct ast_sip_contact *contact = object;
-	struct contact_expiration *expiration;
-	int expires = MAX(0, ast_tvdiff_ms(contact->expiration_time, ast_tvnow()));
+	struct ao2_container *contacts;
+	struct ast_variable *var;
+	char *time = alloca(64);
 
-	if (ast_tvzero(contact->expiration_time)) {
-		return;
-	}
+	while (check_interval) {
+		sleep(check_interval);
 
-	expiration = ao2_alloc_options(sizeof(*expiration), contact_expiration_destroy,
-		AO2_ALLOC_OPT_LOCK_NOLOCK);
-	if (!expiration) {
-		return;
-	}
+		sprintf(time, "%ld", ast_tvnow().tv_sec);
+		var = ast_variable_new("expiration_time <=", time, "");
 
-	expiration->contact = (struct ast_sip_contact*)contact;
-	ao2_ref(expiration->contact, +1);
+		ast_debug(4, "Woke up at %s  Interval: %d\n", time, check_interval);
 
-	ao2_ref(expiration, +1);
-	if ((expiration->sched = ast_sched_add(sched, expires, contact_expiration_expire, expiration)) < 0) {
-		ao2_ref(expiration, -1);
-		ast_log(LOG_ERROR, "Scheduled expiration for contact '%s' could not be performed, contact may persist past life\n",
-			ast_sorcery_object_get_id(contact));
-	} else {
-		ao2_link(contact_autoexpire, expiration);
-	}
-	ao2_ref(expiration, -1);
-}
-
-/*! \brief Observer callback for when a contact is updated */
-static void contact_expiration_observer_updated(const void *object)
-{
-	const struct ast_sip_contact *contact = object;
-	struct contact_expiration *expiration;
-	int expires = MAX(0, ast_tvdiff_ms(contact->expiration_time, ast_tvnow()));
+		contacts = ast_sorcery_retrieve_by_fields(ast_sip_get_sorcery(), "contact",
+			AST_RETRIEVE_FLAG_MULTIPLE, var);
 
-	expiration = ao2_find(contact_autoexpire, ast_sorcery_object_get_id(contact),
-		OBJ_SEARCH_KEY);
-	if (!expiration) {
-		return;
+		ast_variables_destroy(var);
+		if (contacts) {
+			ast_debug(3, "Expiring %d contacts\n\n", ao2_container_count(contacts));
+			ao2_callback(contacts, OBJ_NODATA, expire_contact, NULL);
+			ao2_ref(contacts, -1);
+		}
 	}
 
-	AST_SCHED_REPLACE_UNREF(expiration->sched, sched, expires, contact_expiration_expire,
-		expiration, ao2_cleanup(expiration), ao2_cleanup(expiration), ao2_ref(expiration, +1));
-	ao2_ref(expiration, -1);
+	return NULL;
 }
 
-/*! \brief Observer callback for when a contact is deleted */
-static void contact_expiration_observer_deleted(const void *object)
+static void expiration_global_loaded(const char *object_type)
 {
-	struct contact_expiration *expiration;
-
-	expiration = ao2_find(contact_autoexpire, ast_sorcery_object_get_id(object),
-		OBJ_SEARCH_KEY | OBJ_UNLINK);
-	if (!expiration) {
-		return;
-	}
-
-	AST_SCHED_DEL_UNREF(sched, expiration->sched, ao2_cleanup(expiration));
-	ao2_ref(expiration, -1);
-}
-
-/*! \brief Observer callbacks for autoexpiring contacts */
-static const struct ast_sorcery_observer contact_expiration_observer = {
-	.created = contact_expiration_observer_created,
-	.updated = contact_expiration_observer_updated,
-	.deleted = contact_expiration_observer_deleted,
-};
-
-/*! \brief Callback function which deletes a contact if it has expired or sets up auto-expiry */
-static int contact_expiration_setup(void *obj, void *arg, int flags)
-{
-	struct ast_sip_contact *contact = obj;
-	int expires = MAX(0, ast_tvdiff_ms(contact->expiration_time, ast_tvnow()));
-
-	if (!expires) {
-		ast_sorcery_delete(ast_sip_get_sorcery(), contact);
+	check_interval = ast_sip_get_contact_expiration_check_interval();
+
+	/* Observer calls are serialized so this is safe without it's own lock */
+	if (check_interval) {
+		if (check_thread == AST_PTHREADT_NULL) {
+			if (ast_pthread_create_background(&check_thread, NULL, check_expiration_thread, NULL)) {
+				ast_log(LOG_ERROR, "Could not create thread for checking contact expiration.\n");
+				return;
+			}
+			ast_debug(3, "Interval = %d, starting thread\n", check_interval);
+		}
 	} else {
-		contact_expiration_observer_created(contact);
-	}
-
-	return 0;
-}
-
-/*! \brief Initialize auto-expiration of any existing contacts */
-static void contact_expiration_initialize_existing(void)
-{
-	struct ao2_container *contacts;
-
-	contacts = ast_sorcery_retrieve_by_fields(ast_sip_get_sorcery(), "contact",
-		AST_RETRIEVE_FLAG_MULTIPLE | AST_RETRIEVE_FLAG_ALL, NULL);
-	if (!contacts) {
-		return;
+		if (check_thread != AST_PTHREADT_NULL) {
+			pthread_kill(check_thread, SIGURG);
+			check_thread = AST_PTHREADT_NULL;
+			ast_debug(3, "Interval = 0, shutting thread down\n");
+		}
 	}
-
-	ao2_callback(contacts, OBJ_NODATA, contact_expiration_setup, NULL);
-	ao2_ref(contacts, -1);
 }
 
-static int unload_observer_delete(void *obj, void *arg, int flags)
-{
-	struct contact_expiration *expiration = obj;
-
-	AST_SCHED_DEL_UNREF(sched, expiration->sched, ao2_cleanup(expiration));
-	return CMP_MATCH;
-}
+/*! \brief Observer which is used to update our interval when the global setting changes */
+static struct ast_sorcery_observer expiration_global_observer = {
+	.loaded = expiration_global_loaded,
+};
 
 static int unload_module(void)
 {
-	ast_sorcery_observer_remove(ast_sip_get_sorcery(), "contact", &contact_expiration_observer);
-	if (sched) {
-		ao2_callback(contact_autoexpire, OBJ_MULTIPLE | OBJ_NODATA | OBJ_UNLINK,
-			unload_observer_delete, NULL);
-		ast_sched_context_destroy(sched);
-		sched = NULL;
+	if (check_thread != AST_PTHREADT_NULL) {
+		pthread_kill(check_thread, SIGURG);
+		check_thread = AST_PTHREADT_NULL;
 	}
-	ao2_cleanup(contact_autoexpire);
-	contact_autoexpire = NULL;
+
+	ast_sorcery_observer_remove(ast_sip_get_sorcery(), "global", &expiration_global_observer);
 
 	return 0;
 }
 
+
 static int load_module(void)
 {
 	CHECK_PJSIP_MODULE_LOADED();
 
-	contact_autoexpire = ao2_container_alloc_options(AO2_ALLOC_OPT_LOCK_NOLOCK,
-		CONTACT_AUTOEXPIRE_BUCKETS, contact_expiration_hash, contact_expiration_cmp);
-	if (!contact_autoexpire) {
-		ast_log(LOG_ERROR, "Could not create container for contact auto-expiration\n");
-		return AST_MODULE_LOAD_FAILURE;
-	}
-
-	if (!(sched = ast_sched_context_create())) {
-		ast_log(LOG_ERROR, "Could not create scheduler for contact auto-expiration\n");
-		unload_module();
-		return AST_MODULE_LOAD_FAILURE;
-	}
-
-	if (ast_sched_start_thread(sched)) {
-		ast_log(LOG_ERROR, "Could not start scheduler thread for contact auto-expiration\n");
-		unload_module();
-		return AST_MODULE_LOAD_FAILURE;
-	}
-
-	contact_expiration_initialize_existing();
-
-	if (ast_sorcery_observer_add(ast_sip_get_sorcery(), "contact", &contact_expiration_observer)) {
-		ast_log(LOG_ERROR, "Could not add observer for notifications about contacts for contact auto-expiration\n");
-		unload_module();
-		return AST_MODULE_LOAD_FAILURE;
-	}
+	ast_sorcery_observer_add(ast_sip_get_sorcery(), "global", &expiration_global_observer);
+	ast_sorcery_reload_object(ast_sip_get_sorcery(), "global");
 
 	return AST_MODULE_LOAD_SUCCESS;
 }
diff --git a/res/res_sorcery_astdb.c b/res/res_sorcery_astdb.c
index 45a54946a..e5de9f7bb 100644
--- a/res/res_sorcery_astdb.c
+++ b/res/res_sorcery_astdb.c
@@ -63,65 +63,6 @@ static struct ast_sorcery_wizard astdb_object_wizard = {
 	.close = sorcery_astdb_close,
 };
 
-/*! \brief Helper function which converts from a sorcery object set to a json object */
-static struct ast_json *sorcery_objectset_to_json(const struct ast_variable *objectset)
-{
-	struct ast_json *json = ast_json_object_create();
-	const struct ast_variable *field;
-
-	for (field = objectset; field; field = field->next) {
-		struct ast_json *value = ast_json_string_create(field->value);
-
-		if (!value) {
-			ast_json_unref(json);
-			return NULL;
-		} else if (ast_json_object_set(json, field->name, value)) {
-			ast_json_unref(json);
-			return NULL;
-		}
-	}
-
-	return json;
-}
-
-/*! \brief Helper function which converts a json object to a sorcery object set */
-static struct ast_variable *sorcery_json_to_objectset(struct ast_json *json)
-{
-	struct ast_json_iter *field;
-	struct ast_variable *objset = NULL;
-
-	for (field = ast_json_object_iter(json); field; field = ast_json_object_iter_next(json, field)) {
-		struct ast_json *value = ast_json_object_iter_value(field);
-		struct ast_variable *variable = ast_variable_new(ast_json_object_iter_key(field), ast_json_string_get(value), "");
-
-		if (!variable) {
-			ast_variables_destroy(objset);
-			return NULL;
-		}
-
-		variable->next = objset;
-		objset = variable;
-	}
-
-	return objset;
-}
-
-/*! \brief Helper function which compares two json objects and sees if they are equal, but only looks at the criteria provided */
-static int sorcery_json_equal(struct ast_json *object, struct ast_json *criteria)
-{
-	struct ast_json_iter *field;
-
-	for (field = ast_json_object_iter(criteria); field; field = ast_json_object_iter_next(criteria, field)) {
-		struct ast_json *object_field = ast_json_object_get(object, ast_json_object_iter_key(field));
-
-		if (!object_field || !ast_json_equal(object_field, ast_json_object_iter_value(field))) {
-			return 0;
-		}
-	}
-
-	return 1;
-}
-
 static int sorcery_astdb_create(const struct ast_sorcery *sorcery, void *data, void *object)
 {
 	RAII_VAR(struct ast_json *, objset, ast_sorcery_objectset_json_create(sorcery, object), ast_json_unref);
@@ -144,12 +85,11 @@ static void *sorcery_astdb_retrieve_fields_common(const struct ast_sorcery *sorc
 	const char *prefix = data;
 	char family[strlen(prefix) + strlen(type) + 2];
 	RAII_VAR(struct ast_db_entry *, entries, NULL, ast_db_freetree);
-	RAII_VAR(struct ast_json *, criteria, NULL, ast_json_unref);
 	struct ast_db_entry *entry;
 
 	snprintf(family, sizeof(family), "%s/%s", prefix, type);
 
-	if (!(entries = ast_db_gettree(family, NULL)) || (fields && !(criteria = sorcery_objectset_to_json(fields)))) {
+	if (!(entries = ast_db_gettree(family, NULL))) {
 		return NULL;
 	}
 
@@ -158,14 +98,21 @@ static void *sorcery_astdb_retrieve_fields_common(const struct ast_sorcery *sorc
 		RAII_VAR(struct ast_json *, json, NULL, ast_json_unref);
 		struct ast_json_error error;
 		RAII_VAR(struct ast_variable *, objset, NULL, ast_variables_destroy);
+		RAII_VAR(struct ast_variable *, existing, NULL, ast_variables_destroy);
 		void *object = NULL;
 
 		if (!(json = ast_json_load_string(entry->data, &error))) {
 			return NULL;
-		} else if (criteria && !sorcery_json_equal(json, criteria)) {
+		}
+		if (ast_json_to_ast_variables(json, &existing) != AST_JSON_TO_AST_VARS_CODE_SUCCESS) {
+			return NULL;
+		}
+
+		if (fields && !ast_variable_lists_match(existing, fields, 0)) {
 			continue;
-		} else if (!(objset = sorcery_json_to_objectset(json)) ||
-			!(object = ast_sorcery_alloc(sorcery, type, key)) ||
+		}
+
+		if (!(object = ast_sorcery_alloc(sorcery, type, key)) ||
 			ast_sorcery_objectset_apply(sorcery, object, objset)) {
 			ao2_cleanup(object);
 			return NULL;
@@ -199,9 +146,11 @@ static void *sorcery_astdb_retrieve_id(const struct ast_sorcery *sorcery, void *
 
 	snprintf(family, sizeof(family), "%s/%s", prefix, type);
 
-	if (ast_db_get_allocated(family, id, &value) || !(json = ast_json_load_string(value, &error)) ||
-		!(objset = sorcery_json_to_objectset(json)) || !(object = ast_sorcery_alloc(sorcery, type, id)) ||
-		ast_sorcery_objectset_apply(sorcery, object, objset)) {
+	if (ast_db_get_allocated(family, id, &value)
+		|| !(json = ast_json_load_string(value, &error))
+		|| (ast_json_to_ast_variables(json, &objset) != AST_JSON_TO_AST_VARS_CODE_SUCCESS)
+		|| !(object = ast_sorcery_alloc(sorcery, type, id))
+		|| ast_sorcery_objectset_apply(sorcery, object, objset)) {
 		ast_debug(3, "Failed to retrieve object '%s' from astdb\n", id);
 		ao2_cleanup(object);
 		return NULL;
@@ -310,10 +259,10 @@ static void sorcery_astdb_retrieve_regex(const struct ast_sorcery *sorcery, void
 
 		if (regexec(&expression, key, 0, NULL, 0)) {
 			continue;
-		} else if (!(json = ast_json_load_string(entry->data, &error)) ||
-			!(objset = sorcery_json_to_objectset(json)) ||
-			!(object = ast_sorcery_alloc(sorcery, type, key)) ||
-			ast_sorcery_objectset_apply(sorcery, object, objset)) {
+		} else if (!(json = ast_json_load_string(entry->data, &error))
+			|| (ast_json_to_ast_variables(json, &objset) != AST_JSON_TO_AST_VARS_CODE_SUCCESS)
+			|| !(object = ast_sorcery_alloc(sorcery, type, key))
+			|| ast_sorcery_objectset_apply(sorcery, object, objset)) {
 			regfree(&expression);
 			return;
 		}
diff --git a/res/res_sorcery_config.c b/res/res_sorcery_config.c
index 312015c05..220b5875f 100644
--- a/res/res_sorcery_config.c
+++ b/res/res_sorcery_config.c
@@ -129,7 +129,6 @@ static int sorcery_config_fields_cmp(void *obj, void *arg, int flags)
 {
 	const struct sorcery_config_fields_cmp_params *params = arg;
 	RAII_VAR(struct ast_variable *, objset, NULL, ast_variables_destroy);
-	RAII_VAR(struct ast_variable *, diff, NULL, ast_variables_destroy);
 
 	if (params->regex) {
 		/* If a regular expression has been provided see if it matches, otherwise move on */
@@ -139,11 +138,10 @@ static int sorcery_config_fields_cmp(void *obj, void *arg, int flags)
 		return 0;
 	} else if (params->fields &&
 	    (!(objset = ast_sorcery_objectset_create(params->sorcery, obj)) ||
-	     (ast_sorcery_changeset_create(objset, params->fields, &diff)) ||
-	     diff)) {
+	     (!ast_variable_lists_match(objset, params->fields, 0)))) {
 		/* If we can't turn the object into an object set OR if differences exist between the fields
-	     * passed in and what are present on the object they are not a match.
-	     */
+		 * passed in and what are present on the object they are not a match.
+		 */
 		return 0;
 	}
 
@@ -197,6 +195,7 @@ static void sorcery_config_retrieve_multiple(const struct ast_sorcery *sorcery,
 	if (!config_objects) {
 		return;
 	}
+
 	ao2_callback(config_objects, 0, sorcery_config_fields_cmp, &params);
 }
 
diff --git a/res/res_sorcery_memory.c b/res/res_sorcery_memory.c
index 45bde26f9..e8f603038 100644
--- a/res/res_sorcery_memory.c
+++ b/res/res_sorcery_memory.c
@@ -120,7 +120,6 @@ static int sorcery_memory_fields_cmp(void *obj, void *arg, int flags)
 {
 	const struct sorcery_memory_fields_cmp_params *params = arg;
 	RAII_VAR(struct ast_variable *, objset, NULL, ast_variables_destroy);
-	RAII_VAR(struct ast_variable *, diff, NULL, ast_variables_destroy);
 
 	if (params->regex) {
 		/* If a regular expression has been provided see if it matches, otherwise move on */
@@ -130,8 +129,7 @@ static int sorcery_memory_fields_cmp(void *obj, void *arg, int flags)
 		return 0;
 	} else if (params->fields &&
 	    (!(objset = ast_sorcery_objectset_create(params->sorcery, obj)) ||
-	     (ast_sorcery_changeset_create(objset, params->fields, &diff)) ||
-	     diff)) {
+	     (!ast_variable_lists_match(objset, params->fields, 0)))) {
 		/* If we can't turn the object into an object set OR if differences exist between the fields
 		 * passed in and what are present on the object they are not a match.
 		 */
diff --git a/res/res_sorcery_memory_cache.c b/res/res_sorcery_memory_cache.c
index f2ed5d5c8..4ce4e18e9 100644
--- a/res/res_sorcery_memory_cache.c
+++ b/res/res_sorcery_memory_cache.c
@@ -1251,8 +1251,7 @@ static int sorcery_memory_cache_fields_cmp(void *obj, void *arg, int flags)
 		}
 		return 0;
 	} else if (params->fields &&
-	     (ast_sorcery_changeset_create(cached->objectset, params->fields, &diff) ||
-	     diff)) {
+	     (!ast_variable_lists_match(cached->objectset, params->fields, 0))) {
 		/* If we can't turn the object into an object set OR if differences exist between the fields
 		 * passed in and what are present on the object they are not a match.
 		 */
diff --git a/res/res_sorcery_realtime.c b/res/res_sorcery_realtime.c
index b16069bc0..2c533ea0b 100644
--- a/res/res_sorcery_realtime.c
+++ b/res/res_sorcery_realtime.c
@@ -40,6 +40,18 @@ ASTERISK_FILE_VERSION(__FILE__, "$Revision$")
 /*! \brief They key field used to store the unique identifier for the object */
 #define UUID_FIELD "id"
 
+enum unqualified_fetch {
+	UNQUALIFIED_FETCH_NO,
+	UNQUALIFIED_FETCH_WARN,
+	UNQUALIFIED_FETCH_YES,
+	UNQUALIFIED_FETCH_ERROR,
+};
+
+struct sorcery_config {
+	enum unqualified_fetch fetch;
+	char family[];
+};
+
 static void *sorcery_realtime_open(const char *data);
 static int sorcery_realtime_create(const struct ast_sorcery *sorcery, void *data, void *object);
 static void *sorcery_realtime_retrieve_id(const struct ast_sorcery *sorcery, void *data, const char *type, const char *id);
@@ -66,7 +78,7 @@ static struct ast_sorcery_wizard realtime_object_wizard = {
 
 static int sorcery_realtime_create(const struct ast_sorcery *sorcery, void *data, void *object)
 {
-	const char *family = data;
+	struct sorcery_config *config = data;
 	RAII_VAR(struct ast_variable *, fields, ast_sorcery_objectset_create(sorcery, object), ast_variables_destroy);
 	struct ast_variable *id = ast_variable_new(UUID_FIELD, ast_sorcery_object_get_id(object), "");
 
@@ -79,7 +91,7 @@ static int sorcery_realtime_create(const struct ast_sorcery *sorcery, void *data
 	id->next = fields;
 	fields = id;
 
-	return (ast_store_realtime_fields(family, fields) <= 0) ? -1 : 0;
+	return (ast_store_realtime_fields(config->family, fields) <= 0) ? -1 : 0;
 }
 
 /*! \brief Internal helper function which returns a filtered objectset. 
@@ -149,12 +161,12 @@ static struct ast_variable *sorcery_realtime_filter_objectset(struct ast_variabl
 
 static void *sorcery_realtime_retrieve_fields(const struct ast_sorcery *sorcery, void *data, const char *type, const struct ast_variable *fields)
 {
-	const char *family = data;
+	struct sorcery_config *config = data;
 	RAII_VAR(struct ast_variable *, objectset, NULL, ast_variables_destroy);
 	RAII_VAR(struct ast_variable *, id, NULL, ast_variables_destroy);
 	void *object = NULL;
 
-	if (!(objectset = ast_load_realtime_fields(family, fields))) {
+	if (!(objectset = ast_load_realtime_fields(config->family, fields))) {
 		return NULL;
 	}
 
@@ -178,7 +190,7 @@ static void *sorcery_realtime_retrieve_id(const struct ast_sorcery *sorcery, voi
 
 static void sorcery_realtime_retrieve_multiple(const struct ast_sorcery *sorcery, void *data, const char *type, struct ao2_container *objects, const struct ast_variable *fields)
 {
-	const char *family = data;
+	struct sorcery_config *config = data;
 	RAII_VAR(struct ast_config *, rows, NULL, ast_config_destroy);
 	RAII_VAR(struct ast_variable *, all, NULL, ast_variables_destroy);
 	struct ast_category *row = NULL;
@@ -186,6 +198,18 @@ static void sorcery_realtime_retrieve_multiple(const struct ast_sorcery *sorcery
 	if (!fields) {
 		char field[strlen(UUID_FIELD) + 6], value[2];
 
+		if (config->fetch == UNQUALIFIED_FETCH_NO) {
+			return;
+		}
+		if (config->fetch == UNQUALIFIED_FETCH_ERROR) {
+			ast_log(LOG_ERROR, "Unqualified fetch prevented on %s\n", config->family);
+			return;
+		}
+		if (config->fetch == UNQUALIFIED_FETCH_WARN) {
+			ast_log(LOG_WARNING, "Unqualified fetch attempted on %s\n", config->family);
+			return;
+		}
+
 		/* If no fields have been specified we want all rows, so trick realtime into doing it */
 		snprintf(field, sizeof(field), "%s LIKE", UUID_FIELD);
 		snprintf(value, sizeof(value), "%%");
@@ -197,7 +221,7 @@ static void sorcery_realtime_retrieve_multiple(const struct ast_sorcery *sorcery
 		fields = all;
 	}
 
-	if (!(rows = ast_load_realtime_multientry_fields(family, fields))) {
+	if (!(rows = ast_load_realtime_multientry_fields(config->family, fields))) {
 		return;
 	}
 
@@ -221,16 +245,18 @@ static void sorcery_realtime_retrieve_regex(const struct ast_sorcery *sorcery, v
 	char field[strlen(UUID_FIELD) + 6], value[strlen(regex) + 3];
 	RAII_VAR(struct ast_variable *, fields, NULL, ast_variables_destroy);
 
-	/* The realtime API provides no direct ability to do regex so for now we support a limited subset using pattern matching */
-	snprintf(field, sizeof(field), "%s LIKE", UUID_FIELD);
-	if (regex[0] == '^') {
-		snprintf(value, sizeof(value), "%s%%", regex + 1);
-	} else {
-		snprintf(value, sizeof(value), "%%%s%%", regex);
-	}
+	if (!ast_strlen_zero(regex)) {
+		/* The realtime API provides no direct ability to do regex so for now we support a limited subset using pattern matching */
+		snprintf(field, sizeof(field), "%s LIKE", UUID_FIELD);
+		if (regex[0] == '^') {
+			snprintf(value, sizeof(value), "%s%%", regex + 1);
+		} else {
+			snprintf(value, sizeof(value), "%%%s%%", regex);
+		}
 
-	if (!(fields = ast_variable_new(field, value, ""))) {
-		return;
+		if (!(fields = ast_variable_new(field, value, ""))) {
+			return;
+		}
 	}
 
 	sorcery_realtime_retrieve_multiple(sorcery, data, type, objects, fields);
@@ -238,31 +264,74 @@ static void sorcery_realtime_retrieve_regex(const struct ast_sorcery *sorcery, v
 
 static int sorcery_realtime_update(const struct ast_sorcery *sorcery, void *data, void *object)
 {
-	const char *family = data;
+	struct sorcery_config *config = data;
 	RAII_VAR(struct ast_variable *, fields, ast_sorcery_objectset_create(sorcery, object), ast_variables_destroy);
 
 	if (!fields) {
 		return -1;
 	}
 
-	return (ast_update_realtime_fields(family, UUID_FIELD, ast_sorcery_object_get_id(object), fields) <= 0) ? -1 : 0;
+	return (ast_update_realtime_fields(config->family, UUID_FIELD, ast_sorcery_object_get_id(object), fields) <= 0) ? -1 : 0;
 }
 
 static int sorcery_realtime_delete(const struct ast_sorcery *sorcery, void *data, void *object)
 {
-	const char *family = data;
+	struct sorcery_config *config = data;
 
-	return (ast_destroy_realtime_fields(family, UUID_FIELD, ast_sorcery_object_get_id(object), NULL) <= 0) ? -1 : 0;
+	return (ast_destroy_realtime_fields(config->family, UUID_FIELD, ast_sorcery_object_get_id(object), NULL) <= 0) ? -1 : 0;
 }
 
 static void *sorcery_realtime_open(const char *data)
 {
+	struct sorcery_config *config;
+	char *tmp;
+	char *family;
+	char *option;
+
 	/* We require a prefix for family string generation, or else stuff could mix together */
-	if (ast_strlen_zero(data) || !ast_realtime_is_mapping_defined(data)) {
+	if (ast_strlen_zero(data)) {
+		return NULL;
+	}
+
+	tmp = ast_strdupa(data);
+	family = strsep(&tmp, ",");
+
+	if (!ast_realtime_is_mapping_defined(family)) {
+		return NULL;
+	}
+
+	config = ast_calloc(1, sizeof(*config) + strlen(family) + 1);
+	if (!config) {
 		return NULL;
 	}
 
-	return ast_strdup(data);
+	strcpy(config->family, family); /* Safe */
+	config->fetch = UNQUALIFIED_FETCH_YES;
+
+	while ((option = strsep(&tmp, ","))) {
+		char *name = strsep(&option, "=");
+		char *value = option;
+
+		if (!strcasecmp(name, "allow_unqualified_fetch")) {
+			if (ast_strlen_zero(value) || !strcasecmp(value, "yes")) {
+				config->fetch = UNQUALIFIED_FETCH_YES;
+			} else if (!strcasecmp(value, "no")) {
+				config->fetch = UNQUALIFIED_FETCH_NO;
+			} else if (!strcasecmp(value, "warn")) {
+				config->fetch = UNQUALIFIED_FETCH_WARN;
+			} else if (!strcasecmp(value, "error")) {
+				config->fetch = UNQUALIFIED_FETCH_ERROR;
+			} else {
+				ast_log(LOG_ERROR, "Unrecognized value in %s:%s: '%s'\n", family, name, value);
+				return NULL;
+			}
+		} else {
+			ast_log(LOG_ERROR, "Unrecognized option in %s: '%s'\n", family, name);
+			return NULL;
+		}
+	}
+
+	return config;
 }
 
 static void sorcery_realtime_close(void *data)
diff --git a/tests/test_config.c b/tests/test_config.c
index 3ad57729f..40de2652b 100644
--- a/tests/test_config.c
+++ b/tests/test_config.c
@@ -1672,6 +1672,66 @@ out:
 	return res;
 }
 
+AST_TEST_DEFINE(variable_lists_match)
+{
+	RAII_VAR(struct ast_variable *, left, NULL, ast_variables_destroy);
+	RAII_VAR(struct ast_variable *, right, NULL, ast_variables_destroy);
+	struct ast_variable *var;
+
+	switch (cmd) {
+	case TEST_INIT:
+		info->name = "variable_lists_match";
+		info->category = "/main/config/";
+		info->summary = "Test ast_variable_lists_match";
+		info->description =	"Test ast_variable_lists_match";
+		return AST_TEST_NOT_RUN;
+	case TEST_EXECUTE:
+		break;
+	}
+
+	var = ast_variable_new("aaa", "111", "");
+	ast_test_validate(test, var);
+	left = var;
+	var = ast_variable_new("bbb", "222", "");
+	ast_test_validate(test, var);
+	ast_variable_list_append(&left, var);
+
+	var = ast_variable_new("aaa", "111", "");
+	ast_test_validate(test, var);
+	right = var;
+
+	ast_test_validate(test, ast_variable_lists_match(left, right, 0));
+	ast_test_validate(test, !ast_variable_lists_match(left, right, 1));
+
+	var = ast_variable_new("bbb", "222", "");
+	ast_test_validate(test, var);
+	ast_variable_list_append(&right, var);
+
+	ast_test_validate(test, ast_variable_lists_match(left, right, 0));
+	ast_test_validate(test, ast_variable_lists_match(left, right, 1));
+
+	var = ast_variable_new("ccc >", "333", "");
+	ast_test_validate(test, var);
+	ast_variable_list_append(&right, var);
+
+	ast_test_validate(test, !ast_variable_lists_match(left, right, 0));
+	ast_test_validate(test, !ast_variable_lists_match(left, right, 1));
+
+	var = ast_variable_new("ccc", "444", "");
+	ast_test_validate(test, var);
+	ast_variable_list_append(&left, var);
+
+	ast_test_validate(test, ast_variable_lists_match(left, right, 0));
+	ast_test_validate(test, !ast_variable_lists_match(left, right, 1));
+
+	ast_test_validate(test, !ast_variable_lists_match(left, NULL, 0));
+	ast_test_validate(test, ast_variable_lists_match(NULL, NULL, 0));
+	ast_test_validate(test, !ast_variable_lists_match(NULL, right, 0));
+	ast_test_validate(test, ast_variable_lists_match(left, left, 0));
+
+	return AST_TEST_PASS;
+}
+
 static int unload_module(void)
 {
 	AST_TEST_UNREGISTER(config_basic_ops);
@@ -1682,6 +1742,7 @@ static int unload_module(void)
 	AST_TEST_UNREGISTER(ast_parse_arg_test);
 	AST_TEST_UNREGISTER(config_options_test);
 	AST_TEST_UNREGISTER(config_dialplan_function);
+	AST_TEST_UNREGISTER(variable_lists_match);
 	return 0;
 }
 
@@ -1695,6 +1756,7 @@ static int load_module(void)
 	AST_TEST_REGISTER(ast_parse_arg_test);
 	AST_TEST_REGISTER(config_options_test);
 	AST_TEST_REGISTER(config_dialplan_function);
+	AST_TEST_REGISTER(variable_lists_match);
 	return AST_MODULE_LOAD_SUCCESS;
 }
 
diff --git a/tests/test_sorcery_astdb.c b/tests/test_sorcery_astdb.c
index b87ed74f8..b7566b7ae 100644
--- a/tests/test_sorcery_astdb.c
+++ b/tests/test_sorcery_astdb.c
@@ -298,7 +298,7 @@ AST_TEST_DEFINE(object_retrieve_multiple_field)
 	RAII_VAR(struct ast_sorcery *, sorcery, NULL, deinitialize_sorcery);
 	RAII_VAR(struct test_sorcery_object *, obj, NULL, ao2_cleanup);
 	RAII_VAR(struct ao2_container *, objects, NULL, ao2_cleanup);
-	RAII_VAR(struct ast_variable *, fields, ast_variable_new("joe", "6", ""), ast_variables_destroy);
+	RAII_VAR(struct ast_variable *, fields, ast_variable_new("joe >=", "6", ""), ast_variables_destroy);
 
 	switch (cmd) {
 	case TEST_INIT:
@@ -345,7 +345,7 @@ AST_TEST_DEFINE(object_retrieve_multiple_field)
 	ao2_cleanup(objects);
 	ast_variables_destroy(fields);
 
-	if (!(fields = ast_variable_new("joe", "7", ""))) {
+	if (!(fields = ast_variable_new("joe <", "6", ""))) {
 		ast_test_status_update(test, "Failed to create fields for multiple retrieval\n");
 		return AST_TEST_FAIL;
 	} else if (!(objects = ast_sorcery_retrieve_by_fields(sorcery, "test", AST_RETRIEVE_FLAG_MULTIPLE, fields))) {
diff --git a/tests/test_sorcery_realtime.c b/tests/test_sorcery_realtime.c
index eaee9ffbf..3ed14623e 100644
--- a/tests/test_sorcery_realtime.c
+++ b/tests/test_sorcery_realtime.c
@@ -42,60 +42,12 @@ ASTERISK_FILE_VERSION(__FILE__, "")
 /*! \brief Configuration structure which contains all stored objects */
 static struct ast_config *realtime_objects;
 
-/*! \brief Helper function which finds a given variable */
-static const struct ast_variable *realtime_find_variable(const struct ast_variable *fields, const char *name)
-{
-	const struct ast_variable *variable;
-
-	for (variable = fields; variable; variable = variable->next) {
-		if (!strcmp(variable->name, name)) {
-			return variable;
-		}
-	}
-
-	return NULL;
-}
-
-/*! \brief Helper function which returns if an object is matching or not */
-static int realtime_is_object_matching(const char *object_id, const struct ast_variable *fields)
-{
-	const struct ast_variable *field;
-
-	for (field = fields; field; field = field->next) {
-		char *name = ast_strdupa(field->name), *like;
-		const char *value;
-
-		/* If we are doing a pattern matching we need to remove the LIKE from the name */
-		if ((like = strstr(name, " LIKE"))) {
-			char *field_value = ast_strdupa(field->value);
-
-			*like = '\0';
-
-			value = ast_strdupa(ast_variable_retrieve(realtime_objects, object_id, name));
-
-			field_value = ast_strip_quoted(field_value, "%", "%");
-
-			if (strncmp(value, field_value, strlen(field_value))) {
-				return 0;
-			}
-		} else {
-			value = ast_variable_retrieve(realtime_objects, object_id, name);
-
-			if (ast_strlen_zero(value) || strcmp(value, field->value)) {
-				return 0;
-			}
-		}
-	}
-
-	return 1;
-}
-
 static struct ast_variable *realtime_sorcery(const char *database, const char *table, const struct ast_variable *fields)
 {
 	char *object_id = NULL;
 
 	while ((object_id = ast_category_browse(realtime_objects, object_id))) {
-		if (!realtime_is_object_matching(object_id, fields)) {
+		if (!ast_variable_lists_match(ast_category_root(realtime_objects, object_id), fields, 0)) {
 			continue;
 		}
 
@@ -116,8 +68,9 @@ static struct ast_config *realtime_sorcery_multi(const char *database, const cha
 
 	while ((object_id = ast_category_browse(realtime_objects, object_id))) {
 		struct ast_category *object;
+		const struct ast_variable *object_fields = ast_category_root(realtime_objects, object_id);
 
-		if (!realtime_is_object_matching(object_id, fields)) {
+		if (!ast_variable_lists_match(object_fields, fields, 0)) {
 			continue;
 		}
 
@@ -154,7 +107,7 @@ static int realtime_sorcery_update(const char *database, const char *table, cons
 static int realtime_sorcery_store(const char *database, const char *table, const struct ast_variable *fields)
 {
 	/* The key field is explicit within res_sorcery_realtime */
-	const struct ast_variable *keyfield = realtime_find_variable(fields, "id");
+	const struct ast_variable *keyfield = ast_variable_find_variable_in_list(fields, "id");
 	struct ast_category *object;
 
 	if (!keyfield || ast_category_exist(realtime_objects, keyfield->value, NULL) || !(object = ast_category_new(keyfield->value, "", 0))) {
@@ -201,7 +154,7 @@ static void *test_sorcery_object_alloc(const char *id)
 	return ast_sorcery_generic_alloc(sizeof(struct test_sorcery_object), NULL);
 }
 
-static struct ast_sorcery *alloc_and_initialize_sorcery(void)
+static struct ast_sorcery *alloc_and_initialize_sorcery(char *table)
 {
 	struct ast_sorcery *sorcery;
 
@@ -209,7 +162,7 @@ static struct ast_sorcery *alloc_and_initialize_sorcery(void)
 		return NULL;
 	}
 
-	if ((ast_sorcery_apply_default(sorcery, "test", "realtime", "sorcery_realtime_test") != AST_SORCERY_APPLY_SUCCESS) ||
+	if ((ast_sorcery_apply_default(sorcery, "test", "realtime", table) != AST_SORCERY_APPLY_SUCCESS) ||
 		ast_sorcery_internal_object_register(sorcery, "test", test_sorcery_object_alloc, NULL, NULL) ||
 		!(realtime_objects = ast_config_new())) {
 		ast_sorcery_unref(sorcery);
@@ -246,7 +199,7 @@ AST_TEST_DEFINE(object_create)
 		break;
 	}
 
-	if (!(sorcery = alloc_and_initialize_sorcery())) {
+	if (!(sorcery = alloc_and_initialize_sorcery("sorcery_realtime_test"))) {
 		ast_test_status_update(test, "Failed to open sorcery structure\n");
 		return AST_TEST_FAIL;
 	}
@@ -281,7 +234,7 @@ AST_TEST_DEFINE(object_retrieve_id)
 		break;
 	}
 
-	if (!(sorcery = alloc_and_initialize_sorcery())) {
+	if (!(sorcery = alloc_and_initialize_sorcery("sorcery_realtime_test"))) {
 		ast_test_status_update(test, "Failed to open sorcery structure\n");
 		return AST_TEST_FAIL;
 	}
@@ -344,7 +297,7 @@ AST_TEST_DEFINE(object_retrieve_field)
 		return AST_TEST_FAIL;
 	}
 
-	if (!(sorcery = alloc_and_initialize_sorcery())) {
+	if (!(sorcery = alloc_and_initialize_sorcery("sorcery_realtime_test"))) {
 		ast_test_status_update(test, "Failed to open sorcery structure\n");
 		return AST_TEST_FAIL;
 	}
@@ -402,7 +355,7 @@ AST_TEST_DEFINE(object_retrieve_multiple_all)
 		break;
 	}
 
-	if (!(sorcery = alloc_and_initialize_sorcery())) {
+	if (!(sorcery = alloc_and_initialize_sorcery("sorcery_realtime_test"))) {
 		ast_test_status_update(test, "Failed to open sorcery structure\n");
 		return AST_TEST_FAIL;
 	}
@@ -440,6 +393,63 @@ AST_TEST_DEFINE(object_retrieve_multiple_all)
 	return AST_TEST_PASS;
 }
 
+AST_TEST_DEFINE(object_retrieve_multiple_all_nofetch)
+{
+	RAII_VAR(struct ast_sorcery *, sorcery, NULL, deinitialize_sorcery);
+	RAII_VAR(struct test_sorcery_object *, obj, NULL, ao2_cleanup);
+	RAII_VAR(struct ao2_container *, objects, NULL, ao2_cleanup);
+
+	switch (cmd) {
+	case TEST_INIT:
+		info->name = "object_retrieve_multiple_all_nofetch";
+		info->category = "/res/sorcery_realtime/";
+		info->summary = "sorcery multiple object retrieval unit test";
+		info->description =
+			"Test multiple object retrieval in sorcery using realtime wizard";
+		return AST_TEST_NOT_RUN;
+	case TEST_EXECUTE:
+		break;
+	}
+
+	if (!(sorcery = alloc_and_initialize_sorcery("sorcery_realtime_test,allow_unqualified_fetch=no"))) {
+		ast_test_status_update(test, "Failed to open sorcery structure\n");
+		return AST_TEST_FAIL;
+	}
+
+	if (!(obj = ast_sorcery_alloc(sorcery, "test", "blah"))) {
+		ast_test_status_update(test, "Failed to allocate a known object type\n");
+		return AST_TEST_FAIL;
+	}
+
+	if (ast_sorcery_create(sorcery, obj)) {
+		ast_test_status_update(test, "Failed to create object using realtime wizard\n");
+		return AST_TEST_FAIL;
+	}
+
+	ao2_cleanup(obj);
+
+	if (!(obj = ast_sorcery_alloc(sorcery, "test", "blah2"))) {
+		ast_test_status_update(test, "Failed to allocate second instance of a known object type\n");
+		return AST_TEST_FAIL;
+	}
+
+	if (ast_sorcery_create(sorcery, obj)) {
+		ast_test_status_update(test, "Failed to create second object using realtime wizard\n");
+		return AST_TEST_FAIL;
+	}
+
+	if (!(objects = ast_sorcery_retrieve_by_fields(sorcery, "test", AST_RETRIEVE_FLAG_MULTIPLE | AST_RETRIEVE_FLAG_ALL, NULL))) {
+		ast_test_status_update(test, "Failed to retrieve a container of all objects\n");
+		return AST_TEST_FAIL;
+	} else if (ao2_container_count(objects) != 0) {
+		ast_test_status_update(test, "Received a container with objects in it when there should be none\n");
+		return AST_TEST_FAIL;
+	}
+
+	return AST_TEST_PASS;
+}
+
+
 AST_TEST_DEFINE(object_retrieve_multiple_field)
 {
 	RAII_VAR(struct ast_sorcery *, sorcery, NULL, deinitialize_sorcery);
@@ -464,7 +474,7 @@ AST_TEST_DEFINE(object_retrieve_multiple_field)
 		return AST_TEST_FAIL;
 	}
 
-	if (!(sorcery = alloc_and_initialize_sorcery())) {
+	if (!(sorcery = alloc_and_initialize_sorcery("sorcery_realtime_test"))) {
 		ast_test_status_update(test, "Failed to open sorcery structure\n");
 		return AST_TEST_FAIL;
 	}
@@ -524,7 +534,7 @@ AST_TEST_DEFINE(object_retrieve_regex)
 		break;
 	}
 
-	if (!(sorcery = alloc_and_initialize_sorcery())) {
+	if (!(sorcery = alloc_and_initialize_sorcery("sorcery_realtime_test"))) {
 		ast_test_status_update(test, "Failed to open sorcery structure\n");
 		return AST_TEST_FAIL;
 	}
@@ -574,6 +584,74 @@ AST_TEST_DEFINE(object_retrieve_regex)
 	return AST_TEST_PASS;
 }
 
+AST_TEST_DEFINE(object_retrieve_regex_nofetch)
+{
+	RAII_VAR(struct ast_sorcery *, sorcery, NULL, deinitialize_sorcery);
+	RAII_VAR(struct test_sorcery_object *, obj, NULL, ao2_cleanup);
+	RAII_VAR(struct ao2_container *, objects, NULL, ao2_cleanup);
+
+	switch (cmd) {
+	case TEST_INIT:
+		info->name = "object_retrieve_regex_nofetch";
+		info->category = "/res/sorcery_realtime/";
+		info->summary = "sorcery multiple object retrieval using regex unit test";
+		info->description =
+			"Test multiple object retrieval in sorcery using regular expression for matching using realtime wizard";
+		return AST_TEST_NOT_RUN;
+	case TEST_EXECUTE:
+		break;
+	}
+
+	if (!(sorcery = alloc_and_initialize_sorcery("sorcery_realtime_test,allow_unqualified_fetch=no"))) {
+		ast_test_status_update(test, "Failed to open sorcery structure\n");
+		return AST_TEST_FAIL;
+	}
+
+	if (!(obj = ast_sorcery_alloc(sorcery, "test", "blah-98joe"))) {
+		ast_test_status_update(test, "Failed to allocate a known object type\n");
+		return AST_TEST_FAIL;
+	}
+
+	if (ast_sorcery_create(sorcery, obj)) {
+		ast_test_status_update(test, "Failed to create object using realtime wizard\n");
+		return AST_TEST_FAIL;
+	}
+
+	ao2_cleanup(obj);
+
+	if (!(obj = ast_sorcery_alloc(sorcery, "test", "blah-93joe"))) {
+		ast_test_status_update(test, "Failed to allocate second instance of a known object type\n");
+		return AST_TEST_FAIL;
+	}
+
+	if (ast_sorcery_create(sorcery, obj)) {
+		ast_test_status_update(test, "Failed to create second object using astdb wizard\n");
+		return AST_TEST_FAIL;
+	}
+
+	ao2_cleanup(obj);
+
+	if (!(obj = ast_sorcery_alloc(sorcery, "test", "neener-93joe"))) {
+		ast_test_status_update(test, "Failed to allocate third instance of a known object type\n");
+		return AST_TEST_FAIL;
+	}
+
+	if (ast_sorcery_create(sorcery, obj)) {
+		ast_test_status_update(test, "Failed to create third object using astdb wizard\n");
+		return AST_TEST_FAIL;
+	}
+
+	if (!(objects = ast_sorcery_retrieve_by_regex(sorcery, "test", ""))) {
+		ast_test_status_update(test, "Failed to retrieve a container of objects\n");
+		return AST_TEST_FAIL;
+	} else if (ao2_container_count(objects) != 0) {
+		ast_test_status_update(test, "Received a container with incorrect number of objects in it: %d instead of 0\n", ao2_container_count(objects));
+		return AST_TEST_FAIL;
+	}
+
+	return AST_TEST_PASS;
+}
+
 AST_TEST_DEFINE(object_update)
 {
 	RAII_VAR(struct ast_sorcery *, sorcery, NULL, deinitialize_sorcery);
@@ -592,7 +670,7 @@ AST_TEST_DEFINE(object_update)
 		break;
 	}
 
-	if (!(sorcery = alloc_and_initialize_sorcery())) {
+	if (!(sorcery = alloc_and_initialize_sorcery("sorcery_realtime_test"))) {
 		ast_test_status_update(test, "Failed to open sorcery structure\n");
 		return AST_TEST_FAIL;
 	}
@@ -650,7 +728,7 @@ AST_TEST_DEFINE(object_update_uncreated)
 		break;
 	}
 
-	if (!(sorcery = alloc_and_initialize_sorcery())) {
+	if (!(sorcery = alloc_and_initialize_sorcery("sorcery_realtime_test"))) {
 		ast_test_status_update(test, "Failed to open sorcery structure\n");
 		return AST_TEST_FAIL;
 	}
@@ -685,7 +763,7 @@ AST_TEST_DEFINE(object_delete)
 		break;
 	}
 
-	if (!(sorcery = alloc_and_initialize_sorcery())) {
+	if (!(sorcery = alloc_and_initialize_sorcery("sorcery_realtime_test"))) {
 		ast_test_status_update(test, "Failed to open sorcery structure\n");
 		return AST_TEST_FAIL;
 	}
@@ -732,7 +810,7 @@ AST_TEST_DEFINE(object_delete_uncreated)
 		break;
 	}
 
-	if (!(sorcery = alloc_and_initialize_sorcery())) {
+	if (!(sorcery = alloc_and_initialize_sorcery("sorcery_realtime_test"))) {
 		ast_test_status_update(test, "Failed to open sorcery structure\n");
 		return AST_TEST_FAIL;
 	}
@@ -770,7 +848,7 @@ AST_TEST_DEFINE(object_allocate_on_retrieval)
 		break;
 	}
 
-	if (!(sorcery = alloc_and_initialize_sorcery())) {
+	if (!(sorcery = alloc_and_initialize_sorcery("sorcery_realtime_test"))) {
 		ast_test_status_update(test, "Failed to open sorcery structure\n");
 		return AST_TEST_FAIL;
 	}
@@ -823,7 +901,7 @@ AST_TEST_DEFINE(object_filter)
 		break;
 	}
 
-	if (!(sorcery = alloc_and_initialize_sorcery())) {
+	if (!(sorcery = alloc_and_initialize_sorcery("sorcery_realtime_test"))) {
 		ast_test_status_update(test, "Failed to open sorcery structure\n");
 		return AST_TEST_FAIL;
 	}
@@ -859,8 +937,10 @@ static int unload_module(void)
 	AST_TEST_UNREGISTER(object_retrieve_id);
 	AST_TEST_UNREGISTER(object_retrieve_field);
 	AST_TEST_UNREGISTER(object_retrieve_multiple_all);
+	AST_TEST_UNREGISTER(object_retrieve_multiple_all_nofetch);
 	AST_TEST_UNREGISTER(object_retrieve_multiple_field);
 	AST_TEST_UNREGISTER(object_retrieve_regex);
+	AST_TEST_UNREGISTER(object_retrieve_regex_nofetch);
 	AST_TEST_UNREGISTER(object_update);
 	AST_TEST_UNREGISTER(object_update_uncreated);
 	AST_TEST_UNREGISTER(object_delete);
@@ -879,8 +959,10 @@ static int load_module(void)
 	AST_TEST_REGISTER(object_retrieve_id);
 	AST_TEST_REGISTER(object_retrieve_field);
 	AST_TEST_REGISTER(object_retrieve_multiple_all);
+	AST_TEST_REGISTER(object_retrieve_multiple_all_nofetch);
 	AST_TEST_REGISTER(object_retrieve_multiple_field);
 	AST_TEST_REGISTER(object_retrieve_regex);
+	AST_TEST_REGISTER(object_retrieve_regex_nofetch);
 	AST_TEST_REGISTER(object_update);
 	AST_TEST_REGISTER(object_update_uncreated);
 	AST_TEST_REGISTER(object_delete);
diff --git a/tests/test_strings.c b/tests/test_strings.c
index 5e3446d99..0c69020f8 100644
--- a/tests/test_strings.c
+++ b/tests/test_strings.c
@@ -523,6 +523,68 @@ AST_TEST_DEFINE(escape_test)
 	return AST_TEST_PASS;
 }
 
+AST_TEST_DEFINE(strings_match)
+{
+	switch (cmd) {
+	case TEST_INIT:
+		info->name = "strings_match";
+		info->category = "/main/strings/";
+		info->summary = "Test ast_strings_match";
+		info->description = "Test ast_strings_match";
+		return AST_TEST_NOT_RUN;
+	case TEST_EXECUTE:
+		break;
+	}
+
+	ast_test_validate(test, ast_strings_match("aaa", NULL, "aaa"));
+	ast_test_validate(test, ast_strings_match("aaa", "", "aaa"));
+	ast_test_validate(test, ast_strings_match("aaa", "=", "aaa"));
+	ast_test_validate(test, !ast_strings_match("aaa", "!=", "aaa"));
+	ast_test_validate(test, !ast_strings_match("aaa", NULL, "aba"));
+	ast_test_validate(test, !ast_strings_match("aaa", "", "aba"));
+	ast_test_validate(test, !ast_strings_match("aaa", "=", "aba"));
+	ast_test_validate(test, ast_strings_match("aaa", "!=", "aba"));
+
+	ast_test_validate(test, ast_strings_match("aaa", "<=", "aba"));
+	ast_test_validate(test, ast_strings_match("aaa", "<=", "aaa"));
+	ast_test_validate(test, !ast_strings_match("aaa", "<", "aaa"));
+
+	ast_test_validate(test, !ast_strings_match("aaa", ">=", "aba"));
+	ast_test_validate(test, ast_strings_match("aaa", ">=", "aaa"));
+	ast_test_validate(test, !ast_strings_match("aaa", ">", "aaa"));
+
+	ast_test_validate(test, !ast_strings_match("aaa", "=", "aa"));
+	ast_test_validate(test, ast_strings_match("aaa", ">", "aa"));
+	ast_test_validate(test, !ast_strings_match("aaa", "<", "aa"));
+
+	ast_test_validate(test, ast_strings_match("1", "=", "1"));
+	ast_test_validate(test, !ast_strings_match("1", "!=", "1"));
+	ast_test_validate(test, !ast_strings_match("2", "=", "1"));
+	ast_test_validate(test, ast_strings_match("2", ">", "1"));
+	ast_test_validate(test, ast_strings_match("2", ">=", "1"));
+	ast_test_validate(test, ast_strings_match("2", ">", "1.9888"));
+	ast_test_validate(test, ast_strings_match("2.9", ">", "1"));
+	ast_test_validate(test, ast_strings_match("2", ">", "1"));
+	ast_test_validate(test, ast_strings_match("2.999", "<", "3"));
+	ast_test_validate(test, ast_strings_match("2", ">", "#"));
+
+	ast_test_validate(test, ast_strings_match("abcccc", "like", "%a%c"));
+	ast_test_validate(test, !ast_strings_match("abcccx", "like", "%a%c"));
+	ast_test_validate(test, ast_strings_match("abcccc", "regex", "a[bc]+c"));
+	ast_test_validate(test, !ast_strings_match("abcccx", "regex", "^a[bxdfgtc]+c$"));
+
+	ast_test_validate(test, !ast_strings_match("neener-93joe", "LIKE", "%blah-%"));
+	ast_test_validate(test, ast_strings_match("blah-93joe", "LIKE", "%blah-%"));
+
+	ast_test_validate(test, !ast_strings_match("abcccx", "regex", NULL));
+	ast_test_validate(test, !ast_strings_match("abcccx", NULL, NULL));
+	ast_test_validate(test, !ast_strings_match(NULL, "regex", NULL));
+	ast_test_validate(test, !ast_strings_match(NULL, NULL, "abc"));
+	ast_test_validate(test, !ast_strings_match(NULL, NULL, NULL));
+
+	return AST_TEST_PASS;
+}
+
 static int unload_module(void)
 {
 	AST_TEST_UNREGISTER(str_test);
@@ -531,6 +593,7 @@ static int unload_module(void)
 	AST_TEST_UNREGISTER(strsep_test);
 	AST_TEST_UNREGISTER(escape_semicolons_test);
 	AST_TEST_UNREGISTER(escape_test);
+	AST_TEST_UNREGISTER(strings_match);
 	return 0;
 }
 
@@ -542,6 +605,7 @@ static int load_module(void)
 	AST_TEST_REGISTER(strsep_test);
 	AST_TEST_REGISTER(escape_semicolons_test);
 	AST_TEST_REGISTER(escape_test);
+	AST_TEST_REGISTER(strings_match);
 	return AST_MODULE_LOAD_SUCCESS;
 }