Change-Id: Ie8a9ae3d13bdeaacafc8d28271adc6707f633a5f

This patch adds unit tests for res_http_media cache, that covers nominal
creation and retrieval - and through them as well, staleness and deletion
checks. In addition, this patch adds tests that covers the interaction of
various HTTP headers, including Expires, Etag, and Cache-Control.

ASTERISK-25654

Change-Id: I2db101e307c863857fe416d6f5bf4cace9ac7cf5

This patch is part of a series to resolve deadlocks in chan_sip.c.

* Updated sched unit test to check new behavior.

ASTERISK-25023

Change-Id: Ib69437327b3cda5e14c4238d9ff91b2531b34ef3

The threadpool_auto_increment test fails infrequently for a couple of
reasons
* The threadpool listener was notified of fewer tasks being pushed than
  were actually pushed
* The "was_empty" flag was set to an unexpected value.

The problem is that the test pushes three tasks into the threadpool.
Test expects the threadpool to essentially gather those three tasks, and
then distribute those to the threadpool threads. It also expects that as
the tasks are pushed in, the threadpool listener is alerted immediately
that the tasks have been pushed. In reality, a task can be distributed
to the threadpool threads quicker than expected, meaning that the
threadpool has already emptied by the time each subsequent task is
pushed. In addition, the internal threadpool queue can be delayed so
that the threadpool listener is not alerted that a task has been pushed
even after the task has been executed.

From the test's point of view, there's no way to be able to predict
exactly the order that task execution/listener notifications will occur,
and there is no way to know which listener notifications will indicate
that the threadpool was previously empty.

For this reason, the test has been updated to only check the things it
can check. It ensures that all tasks get executed, that the threads go
idle after the tasks are executed, and that the listener is told the
proper number of tasks that were pushed.

Change-Id: I7673120d74adad64ae6894594a606e102d9a1f2c

When terminating the threads thrashing a sorcery memory cache each
would be told to stop and then we would wait on them. During at
least one thrashing test this was problematic due to the specific
usage pattern in use. It would take some time for termination of the
thread to occur.

This would occur due to contention between the threads retrieving
and the threads updating the cache. As the retrieving threads are
given priority it may be some time before the updating threads
are able to proceed.

This change makes it so all threads are told to stop and then each
are joined to ensure they stop. This way all the threads should
stop at around the same time instead of waiting for one to stop,
the next to stop, then the next, and so on. As a result of this
the execution time for each thrash test is much closer to their
expected value than previously seen as well.

Change-Id: I04a53470b0ea4170b8819180b0bd7475f3642827

test_dlinklists doesn't need to NOTICE everyone that every macro worked.

res_phoneprov doesn't need to VERBOSE everyone that a phoneprov extension or
provider was registered.

res_odbc was missing a newline at the end of one message.

Change-Id: I6c06361518ef3711821795e535acd439782a995e

This change makes the thread_timeout_thrash unit test wait for
each task to complete. This fixes the problem where the test would
prematurely end when all threads were gone and a new one had to be
started to handle the last task. It also increases the thrasing as
it is now more likely for each task to encounter the above scenario.

This also fixes a memory leak where the data for each task was not
being freed.

ASTERISK-25611 #close

Change-Id: I5017d621a4dc911f509074c16229b86bff2fb3c6

res_sorcery_realtime's search-by-regex callback performed a check to
ensure that the passed-in regex began with a caret (^). If it did not,
then no results would be returned.

This callback only started to become used when "like" support was added
to PJSIP CLI commands. The CLI command for listing objects would pass an
empty regex ("") to the sorcery backend if no "like" statement was
present. For most sorcery backends, this resulted in returning all
objects. However, for realtime, this resulted in returning no objects.

This commit seeks to fix the regression by removing the requirement from
res_sorcery_realtime for the passed-in-regex to begin with a caret.

ASTERISK-25689 #close
Reported by Marcelo Terres

Change-Id: I22b4dc5d7f3f11bb29ac2e42ef94682e9bab3b20

The cache_clear test was written to expect duplicate Stasis messages
sent from the technology endpoint to the all caching topic. This patch
fixes the test to no longer expect these duplicate messages.

ASTERISK-25137

Change-Id: I58075d70d6cdf42e792e0fb63ba624720bfce981

When appending all formats of a type all the codecs are iterated
and added. This operation was incorrectly adding the ast_format_none
format which is special in that it is supposed to be used when no
format is present. It shouldn't be appended.

ASTERISK-25535

Change-Id: I7b00f3bdf4a5f3022e483d6ece602b1e8b12827c

This change adds handling of dead worker threads when moving them
to be active. When this happens the worker thread is removed from
both the active and idle threads container. If no threads are able
to be moved to active then the pool grows as configured.

A unit test has also been added which thrashes the idle timeout
and thread activation to exploit any race conditions between the
two.

ASTERISK-25546 #close

Change-Id: I6c455f9a40de60d9e86458d447b548fb52ba1143

ASTERISK-25533 #close

Change-Id: Ie1a9d1a6511b3f1a56b93d04475fbf8a4e40010a

Some codecs that may be a third party library to Asterisk need to have
knowledge of the format attributes that were negotiated. Unfortunately,
when the great format migration of Asterisk 13 occurred, that ability
was lost.

This patch adds an API call, ast_format_attribute_get, to the core
format API, along with updates to the unit test to check the new API
call. A new callback is also now available for format attribute modules,
such that they can provide the format attribute values they manage.

Note that the API returns a void *. This is done as the format attribute
modules themselves may store format attributes in any particular manner
they like. Care should be taken by consumers of the API to check the
return value before casting and dereferencing. Consumers will obviously
need to have a priori knowledge of the type of the format attribute as
well.

Change-Id: Ieec76883dfb46ecd7aff3dc81a52c81f4dc1b9e3

clock_gettime() is, unfortunately, not portable. But I did like that
over our usual `ts.tv_nsec = tv.tv_usec * 1000` copy/paste code we
usually do when we want a timespec and all we have is ast_tvnow().

This patch adds ast_tsnow(), which mimics ast_tvnow(), but returns a
timespec. If clock_gettime() is available, it will use that. Otherwise
ast_tsnow() falls back to using ast_tvnow().

Change-Id: Ibb1ee67ccf4826b9b76d5a5eb62e90b29b6c456e

A testsuite test recently failed due to a crash that occurred in the DNS
core. The problem was that the test could not resolve an address, did
not set a result on the DNS query, and then indicated the query was
completed. The DNS core does not handle the case of a query with no
result gracefully, and so there is a crash.

This changeset makes the DNS system resolver set a result with a
zero-length answer in the case that a DNS resolution failure occurs
early. The DNS core now also will accept such a response without
treating it as invalid input. A unit test was updated to no longer treat
setting a zero-length response as off-nominal.

Change-Id: Ie56641e22debdaa61459e1c9a042e23b78affbf6

Fixes for issues with the ASTERISK-24934 patch.

* Fixed ast_escape_alloc() and ast_escape_c_alloc() if the s parameter is
an empty string.  If it were an empty string the functions returned NULL
as if there were a memory allocation failure.  This failure caused the AMI
VarSet event to not get posted if the new value was an empty string.

* Fixed dest buffer overwrite potential in ast_escape() and
ast_escape_c().  If the dest buffer size is smaller than the space needed
by the escaped s parameter string then the dest buffer would be written
beyond the end by the nul string terminator.  The num parameter was really
the dest buffer size parameter so I renamed it to size.

* Made nul terminate the dest buffer if the source string parameter s was
an empty string in ast_escape() and ast_escape_c().

* Updated ast_escape() and ast_escape_c() doxygen function description
comments to reflect reality.

* Added some more unit test cases to /main/strings/escape to cover the
empty source string issues.

ASTERISK-25255 #close
Reported by: Richard Mudgett

Change-Id: Id77fc704600ebcce81615c1200296f74de254104

This patch adds a new API to the Asterisk core that acts as a media
cache. The core API itself is mostly a thin wrapper around some bucket
API provided implementation that itself acts as the mechanism of
retrieval for media. The media cache API in the core provides the
following:
 * A very thin in-memory cache of the active bucket_file items. Unlike a
   more traditional cache, it provides no expiration mechanisms. Most
   queries that hit the in-memory cache will also call into the bucket
   implementations as well. The bucket implementations are responsible
   for determining whether or not the active record is active and valid.
   This makes sense for the most likely implementation of a media cache
   backend, i.e., HTTP. The HTTP layer itself is the actual arbiter of
   whether or not a record is truly active; as such, the in-memory cache
   in the core has to defer to it.
 * The ability to create new items in the media cache from local
   resources. This allows for re-creation of items in the cache on
   restart.
 * Synchronization of items in the media cache to the AstDB. This
   also includes various pieces of important metadata.

The API provides sufficient access that higher level APIs, such as the
file or app APIs, do not have to worry about the semantics of the bucket
APIs when needing to playback a resource.

In addition, this patch provides unit tests for the media cache API. The
unit tests use a fake bucket backend to verify correctness.

Change-Id: I11227abbf14d8929eeb140ddd101dd5c3820391e

This patch adds more tests that exercise the device state API. This includes:

* Tests that cover adding a device state provider, as well as deleting a
  device state provider. This also verifies that you cannot add an
  already added device state provider, and cannot delete an already
  deleted device state provider.
* A test that covers changing device state and receiving said updates
  from a device state subscriber. This also covers hitting both the
  device state cache as well as a custom device state provider.
* A test that covers converting device state to channel state and device
  state values to a string representation and back.
* A test that covers obtaining device state from an active channel and a
  channel driver that provides its own device state.

Change-Id: I2adca67ffb405cd8625a5d6df1e3f9b3d945c08d

This patch enhances the bucket API in two ways.

First, since ast_bucket and ast_bucket_file instances are immutable, a 'clone'
operation has been added that provides a 'clone' of an existing
ast_bucket/ast_bucket_file object. Note that this makes use of the
ast_sorcery_copy operation, along with the copy callback handler on the
"bucket" and "file" object types for the bucket sorcery instance.

Second, there is a need for the bucket API to ask a wizard if an object
is stale. This is particularly useful with the upcoming media cache
enhancements, where we want to ask the backing data storage if the
object we are currently operating on has known updates. This patch adds
API calls for ast_bucket and ast_bucket_file objects, which callback
into their respective sorcery wizards via the sorcery API.

Unit tests have also been added to cover the respective
ast_bucket/ast_bucket_file clone and staleness operations.

Change-Id: Ib0240ba915ece313f1678a085a716021d75d6b4a

This patch enhances the sorcery API to allow for sorcery wizards to
determine if an object is stale. This includes the following:

* Sorcery objects now have a timestamp that is set on creation. Since
  sorcery objects are immutable, this can be used by sorcery wizards to
  determine if an object is stale.

* A new API call has been added, ast_sorcery_is_stale. This API call
  queries the wizards associated with the object, calling a new callback
  function 'is_stale'. Note that if a wizard does not support the new
  callback, objects are always assumed to not be stale.

* Unit tests have been added that cover the new API call.

Change-Id: Ica93c6a4e8a06c0376ea43e00cf702920b806064

Analyzing the code shows that the unit test summary and description
strings should not end with a new-line character.  Where these strings are
used in the code a new-line is provided for output.

Change-Id: I2f4f37988ec363c8d1c5077a2fc8ca841c5cd30c

Analyzing the code shows that the unit test summary and description
strings should not end with a new-line character.  Where these strings are
used in the code a new-line is provided for output.

Change-Id: I129284f5e7ca93d82532334076da4c462d3d9fba

Change-Id: Icf5f13c8e1c2c92a4473bb573ed2dd856ce1b64e

* Fix query_set destruction before we are done kicking the queries off.

* Fixed no queries requested handling.

* Add empty queries request unit test.

* Added missing allocation check in ast_dns_query_set_add().

* Made initial pjsip resolving query vector slightly larger.

ASTERISK-25115
Reported by: John Bigelow

Change-Id: Ie8be8347d0992e93946d72b6e7b1299727b038f2

Those trailing newlines mess up test formatting.

Change-Id: I5e3f3a55b82c9d7acb9661201d4993d1958f1185

Although ast_context_find, ast_context_find_or_create and
ast_context_destroy perform locking of the contexts table,
any context pointer can become invalid at any time that the
contexts table is unlocked. This change adds locking around
all complete operations involving these functions.

Places where ast_context_find was followed by ast_context_destroy
have been replaced with calls ast_context_destroy_by_name.

ASTERISK-25094 #close
Reported by: Corey Farrell

Change-Id: I1866b6787730c9c4f3f836b6133ffe9c820734fa

So this issue is a bit complicated. Since it is possible to pass values to AMI
that contain a '\r\n' (or other similar sequences) these values need to be
escaped. One way to solve this is to escape the values and then pass the escaped
values to the AMI variable parameter string building function. However, this
puts the onus on the pre-build function to escape all string values. This
potentially requires a fair amount of changes along with a lot of string
allocations/freeing for all values.

Surely there is a way to push this complexity down a level into the string
building function itself? This of course is possible, but ends up requiring a
way to distinguish between strings that need to be escaped and those that don't.
The best way to handle this is by introducing a new format specifier in the
format string. For instance a %s (no escape) and %S (escape). However, that is
a bit weird and unexpected.

So faced with those possibilities this patch implements a limited version of the
first option. Instead of attempting to escape all string values this patch only
escapes those values that make sense. This approach limits the number of changes
and doesn't suffer from the odd format specifier problem.

ASTERISK-24934 #close
Reported by: warren smith

Change-Id: Ib55a5b84fe0481b0f2caaaab68c566f392c0aac0

This change adds a CLI command which can perform memory cache thrashing as well
as unit tests which perform thrashing under the following configurations:

1. Low number of unique objects that go stale after 1 second
2. Low number of unique objects that expire after 1 second
3. Low number of unique objects which are constantly updated
4. Large number of unique objects which exceed a defined cache size
5. Large number of unique objects which exceed a defined cache size
   that also expire and go stale rapidly
6. Large number of unique objects which expire and go stale rapidly
7. Large number of unique objects

For all of the above there are a large number of threads constantly
attempting to retrieve random objects and each test runs for a few
seconds.

ASTERISK-25067
Reported by: Matt Jordan

Change-Id: I8c8ceff977332c80ed4a31f10d694d48552b2f78

This patch fixes a number of errors and warning messages in the doxygen
log. Specifically, it addresses:
* A number of files incorrectly places a '\brief' tag immediately after
  a '\file' tag. Doing so emits a warning, as '\file' takes an optional
  argument specifying which file the doxygen comment is for. As '\brief'
  is not a file, doxygen was unamused.
* A grouping of Stasis Topics and Messages in rtp_engine.h was
  incorrectly terminated. We now correctly terminate the grouping, which
  prevents members of rtp_engine.h from showing up in the wrong group.
* Group indicators which are not part of the Stasis Topics and Messages
  group were removed. Group indicators without an \addtogroup or
  \ingroup have no meaning.

Change-Id: Ia1415ffec6767e27233ae1cae5ed5970de5656d4

The config wizard was always pulling the first occurrence of
a variable from an ast_variable list but this gets the template
value from the list instead of any overridden value.  This patch
creates ast_variable_find_last_in_list() in config.c and updates
res_pjsip_config_wizard to use it instead of
ast_variable_find_in_list.  Now the overridden values, where they
exist, are used instead of template variables.

Updated test_config to test the new API.

ASTERISK-25089 #close

Reported-by: George Joseph <george.joseph@fairview5.com>
Tested-by: George Joseph <george.joseph@fairview5.com>
Change-Id: Ifa7ddefc956a463923ee6839dd1ebe021c299de4

These modules save a pointer to the context they create on load, and
use that pointer to destroy the context at unload.  It is not safe
to save this pointer, it is replaced during load of pbx_config,
pbx_lua or pbx_ael.

This change causes the modules to pass NULL to ast_context_destroy,
a safer way to perform the unregistration since it does not use
a pointer that could become invalid.

ASTERISK-25085 #close
Reported by: Corey Farrell

Change-Id: I6a00ec8e38046058f97dc703e1adcde9bf517835

Change-Id: Icf88f9f861c6b2a16e5f626ff25795218a6f2723

Currently you can 'apply' a wizard to an object type but the wizard
always goes at the end of the object type's wizard list.  This patch
adds a new ast_sorcery_insert_wizard_mapping function that allows
you to insert a wizard anyplace in the list.  I.E.  You could
add a caching wizard to an object type and place it before all
wizards.

ast_sorcery_get_wizard_mapping_count and
ast_sorcery_get_wizard_mapping were added to allow examination
of the mapping list.

ast_sorcery_remove_mapping was added to remove a mapping by name.

As part of this patch, the object type's wizard list was converted
from an ao2_container to an AST_VECTOR_RW.

A new test was added to test_sorcery for this capability.

ASTERISK-25044 #close

Change-Id: I9d2469a9296b2698082c0989e25e6848dc403b57

Based on feedback from Corey Farrell and Y Ateya, a few new
macros have been added...

AST_VECTOR_REMOVE which takes a parameter to indicate if
order should be preserved.

AST_VECTOR_ADD_SORTED which adds an element to
a sorted vector.

AST_VECTOR_RESET which cleans all elements from the vector
leaving the storage intact.

Change-Id: I41d32dbdf7137e0557134efeff9f9f1064b58d14

After using the new vector stuff for real I found...

A bug in AST_VECTOR_INSERT_AT that could cause a seg fault.

The callbacks needed to be closer to ao2_callback in behavior
WRT to CMP_MATCH and CMP_STOP behavior and the ability to return
a vector of matched entries.

A pre-existing issue with APPEND and REPLACE was also fixed.

I also added a new macro to test.h that acts like ast_test_validate
but also accepts a return code variable and a cleanup label.  As well
as printing the error, it sets the rc variable to AST_TEST_FAIL and
does a goto to the specified label on error.  I had a local version
of this in test_vector so I just moved it.

ASTERISK-25045

Change-Id: I05e5e47fd02f61964be13b7e8942bab5d61b29cc

Renamed AST_VECTOR_INSERT to AST_VECTOR_REPLACE because it really
does replace not insert.  The few users of AST_VECTOR_INSERT were
refactored.  Because these are macros, there should be no ABI
compatibility issues.

Added AST_VECTOR_INSERT_AT that actually inserts an element into the
vector at a specific index pushing existing elements to the right.

Added AST_VECTOR_GET_CMP that can retrieve from the vector based
on a user-provided compare function.

Added AST_VECTOR_CALLBACK function that will execute a function
for each element in the vector.  Similar to ao2_callback and
ao2_callback_data functions although the vector callback can take
a variable number of arguments.  This should allow easy migration
to a vector where a container might be too heavy.

Added read/write locked vector and lock manipulation macros.

Added unit tests.

ASTERISK-25045 #close

Change-Id: I2e07ecc709d2f5f91bcab8904e5e9340609b00e0

This function allows code to run ao2_ref against the real
object associated with a weakproxy.  It is useful when
all of the following conditions are true:
* You have a pointer to weakproxy.
* You do not have or need a pointer to the real object.
* You need to ensure the real object exists and is not
  destroyed during a process.

In this case it's wasteful to store a pointer to the real
object just for the sake of releasing it later.

Change-Id: I38a319b83314de75be74207a8771aab269bcca46

The query set documentation states that upon completion queries can be
retrieved for the lifetime of the query set. This is a reasonable
expectation but does not currently occur. This was originally done
to resolve a circular reference between queries and query sets, but
in practice the query can be kept.

This change makes it so a query does not have a reference to the
query set until it begins resolving. It also makes it so that the
reference is given up upon the query being completed. This allows
the queries to remain for the lifetime of the query set. As the
query set on the query is only useful to the query set functionality
and only for the lifetime that the query is resolving this is safe
to do.

ASTERISK-24994 #close
Reported by: Joshua Colp

Change-Id: I54e09c0cb45475896654e7835394524e816d1aa0

This change adds the following:

1. A query set implementation. This is an API that allows queries to be executed in parallel and once all have completed a callback is invoked.
2. Unit tests for the query set implementation.
3. An external PJSIP resolver which uses the DNS core API to do NAPTR, SRV, AAAA, and A lookups.

For the resolver it will do NAPTR, SRV, and AAAA/A lookups in parallel. If NAPTR or SRV
are available it will then do more queries. And so on. Preference is NAPTR > SRV > AAAA/A,
with IPv6 preferred over IPv4. For transport it will prefer TLS > TCP > UDP if no explicit
transport has been provided. Configured transports on the system are taken into account to
eliminate resolved addresses which have no hope of completing.

ASTERISK-24947 #close
Reported by: Joshua Colp

Change-Id: I56cb03ce4f9d3d600776f36928e0b3e379b5d71e

Update test_astobj2_weaken to use the new AST_REGISTER_FILE macro.

Change-Id: Ieedadf16610f2e042f393e0501a36447cd07f83d