There were a number of places in the res_pjsip stack that were getting
all endpoints or all aors, and then filtering them locally.

A good example is pjsip_options which, on startup, retrieves all
endpoints, then the aors for those endpoints, then tests the aors to see
if the qualify_frequency is > 0.  One issue was that it never did
anything with the endpoints other than retrieve the aors so we probably
could have skipped a step and just retrieved all aors. But nevermind.

This worked reasonably well with local config files but with a realtime
backend and thousands of objects, this was a nightmare.  The issue
really boiled down to the fact that while realtime supports predicates
that are passed to the database engine, the non-realtime sorcery
backends didn't.

They do now.

The realtime engines have a scheme for doing simple comparisons. They
take in an ast_variable (or list) for matching, and the name of each
variable can contain an operator.  For instance, a name of
"qualify_frequency >" and a value of "0" would create a SQL predicate
that looks like "where qualify_frequency > '0'".  If there's no operator
after the name, the engines add an '=' so a simple name of
"qualify_frequency" and a value of "10" would return exact matches.

The non-realtime backends decide whether to include an object in a
result set by calling ast_sorcery_changeset_create on every object in
the internal container.  However, ast_sorcery_changeset_create only does
exact string matches though so a name of "qualify_frequency >" and a
value of "0" returns nothing because the literal "qualify_frequency >"
doesn't match any name in the objset set.

So, the real task was to create a generic string matcher that can take a
left value, operator and a right value and perform the match. To that
end, strings.c has a new ast_strings_match(left, operator, right)
function.  Left and right are the strings to operate on and the operator
can be a string containing any of the following: = (or NULL or ""), !=,
>, >=, <, <=, like or regex.  If the operator is like or regex, the
right string should be a %-pattern or a regex expression.  If both left
and right can be converted to float, then a numeric comparison is
performed, otherwise a string comparison is performed.

To use this new function on ast_variables, 2 new functions were added to
config.c.  One that compares 2 ast_variables, and one that compares 2
ast_variable lists.  The former is useful when you want to compare 2
ast_variables that happen to be in a list but don't want to traverse the
list.  The latter will traverse the right list and return true if all
the variables in it match the left list.

Now, the backends' fields_cmp functions call ast_variable_lists_match
instead of ast_sorcery_changeset_create and they can now process the
same syntax as the realtime engines.  The realtime backend just passes
the variable list unaltered to the engine.  The only gotcha is that
there's no common realtime engine support for regex so that's been noted
in the api docs for ast_sorcery_retrieve_by_fields.

Only one more change to sorcery was done...  A new config flag
"allow_unqualified_fetch" was added to reg_sorcery_realtime.
"no": ignore fetches if no predicate fields were supplied.
"error": same as no but emit an error. (good for testing)
"yes": allow (the default);
"warn": allow but emit a warning. (good for testing)

Now on to res_pjsip...

pjsip_options was modified to retrieve aors with qualify_frequency > 0
rather than all endpoints then all aors.  Not only was this a big
improvement in realtime retrieval but even for config files there's an
improvement because we're not going through endpoints anymore.

res_pjsip_mwi was modified to retieve only endpoints with something in
the mailboxes field instead of all endpoints then testing mailboxes.

res_pjsip_registrar_expire was completely refactored.  It was retrieving
all contacts then setting up scheduler entries to check for expiration.
Now, it's a single thread (like keepalive) that periodically retrieves
only contacts whose expiration time is < now and deletes them.  A new
contact_expiration_check_interval was added to global with a default of
30 seconds.

Ross Beer reports that with this patch, his Asterisk startup time dropped
from around an hour to under 30 seconds.

There are still objects that can't be filtered at the database like
identifies, transports, and registrations.  These are not going to be
anywhere near as numerous as endpoints, aors, auths, contacts however.

Back to allow_unqualified_fetch.  If this is set to yes and you have a
very large number of objects in the database, the pjsip CLI commands
will attempt to retrive ALL of them if not qualified with a LIKE.
Worse, if you type "pjsip show endpoint <tab>" guess what's going to
happen? :)  Having a cache helps but all the objects will have to be
retrieved at least once to fill the cache.  Setting
allow_unqualified_fetch=no prevents the mass retrieve and should be used
on endpoints, auths, aors, and contacts.  It should NOT be used for
identifies, registrations and transports since these MUST be
retrieved in bulk.

Example sorcery.conf:

[res_pjsip]
endpoint=config,pjsip.conf,criteria=type=endpoint
endpoint=realtime,ps_endpoints,allow_unqualified_fetch=error

ASTERISK-25826 #close
Reported-by: Ross Beer
Tested-by: Ross Beer

Change-Id: Id2691e447db90892890036e663aaf907b2dc1c67

Blind transfers to a recognized parking extension need to use the parker's
channel variable values to create the dynamic parking lot.  This is
because there is always only one parker while the parkee may actually be a
multi-party bridge.  A multi-party bridge can never supply the needed
channel variables to create the dynamic parking lot.  In the multi-party
bridge blind transfer scenario, the parker's CHANNEL(parkinglot) value and
channel variables are inherited by the local channel used to park the
bridge.

* In park_common_setup(), make use the parker instead of the parkee to
supply the dynamic parking lot channel variable values.  In all but one
case, the parkee is the same as the parker.  However, in the recognized
parking extension blind transfer scenario for a two party bridge they are
different channels.  For consistency, we need to use the parker channel.

* In park_local_transfer(), pass the CHANNEL(parkinglot) value to the
local channel when blind transferring a multi-party bridge to a recognized
parking extension.

* When a local channel starts a call, the Local;2 side needs to inherit
the CHANNEL(parkinglot) value from Local;1.

The DTMF one-touch parking case wasn't even trying to create dynamic
parking lots before it aborted the attempt.

* In parking_park_call(), add missing code to create a dynamic parking
lot.

A DTMF bridge hook is documented as returning -1 to remove the hook.
Though the hook caller is really coded to accept non-zero.  See the
ast_bridge_hook_callback typedef.

* In feature_park_call(), don't remove the DTMF one-touch parking hook
because of an error.

ASTERISK-24605 #close
Reported by:  Philip Correia
Patches:
      call_park.patch (license #6672) patch uploaded by Philip Correia

Change-Id: I221d3a8fcc181877a1158d17004474d35d8016c9

Change-Id: I8f7a8890aef27824301c642d4d15407ac83e6f02

* Remove duplicate res_parking.conf courtesytone config option
documentation.

ASTERISK-24596 #close
Reported by:  Philip Correia

ASTERISK-24605
Reported by:  Philip Correia
Patches:
      call_park_app_doc.patch (license #6672) patch uploaded by Philip Correia

Change-Id: I90a92a891c6494dc08173e675856afcc4764c5b5

The file playback system will now query the media cache and then
the old file functionality. Under normal conditions this will result
in the cache failing to retrieve a file causing a warning message
to get output each time a file is played back.

This change demotes this warning to a debug message.

Change-Id: Ib72246ba300b5cce32774bfb3c26634bfb708624

During stress testing, we have frequently seen crashes occur because a
CLI or AMI command attempts to access information that is in the process
of being destroyed.

When addressing how to fix this issue, we initially considered fixing
individual crashes we observed. However, the changes required to fix
those problems would introduce considerable overhead to the nominal
case. This is not reasonable in order to prevent a crash from occurring
while Asterisk is already shutting down.

Instead, this change makes it so AMI and CLI commands cannot be executed
if Asterisk is being shut down. For AMI, this is absolute. For CLI,
though, certain commands can be registered so that they may be run
during Asterisk shutdown.

ASTERISK-25825 #close

Change-Id: I8887e215ac352fadf7f4c1e082da9089b1421990

The configuration unsigned integer option handler sets flags for the
parser as if the option should be a signed integer (PARSE_INT32),
leading to errors on "out of range" values. Fix flags (PARSE_UINT32).

A fix to res_pjsip is also present which stops invalid flags from
being passed when registering sorcery object fields for qualify
status.

ASTERISK-25612 #close

Change-Id: I96b539336275e0e72a8e8033487d2c3344debd3e

The log message when a MusicOnHold music class was not found was changed
from debug level to WARNING level in Asterisk 11.19 and 13.5.  For those
using realtime musiconhold, this message is wrong because it warns
before checking the database.

This changeset delays the warning until after the database has been
checked.

Reported-by: Conrad de Wet
ASTERISK-25444 #close

Change-Id: I6cfb2db2f9cfbd2bb3d30566ecae361c4abf6dbf

The fix to ASTERISK-25407 introduced the usage of LOG_MAKEPRI. However
this macro is broken in older glibc (< 2.17); it would left-shift the
facility a second time, causing the resultant priority to become
invalid.

The syslog manpage mentions nothing about LOG_MAKEPRI and suggests this:

    The priority argument is formed by ORing the facility and the level
    values [...].

ASTERISK-25510 #close
Reported by: Michael Newton

Change-Id: Ia89debe7fac5ad090c7ef595c0707f31bb1e3d03

Change-Id: Ie8a9ae3d13bdeaacafc8d28271adc6707f633a5f

There is a little known feature in app_controlplayback that will cause the
specified offset to be used relative to the end of a file if a ':end' is
detected within the filename.

This feature is pretty bad, but okay.

However, a bug exists in this code where a ':' detected in the filename
will cause the end pointer to be non-NULL, even if the full ':end' isn't
specified. This causes us to treat an unspecified offset (0) as being
"start playing from the end of the file", resulting in no file playback
occurring.

This patch fixes this bug by resetting the end pointer if ':end' is not
found in the filename.

Change-Id: Ib4c7b1b45283e4effd622a970055c51146892f35

This patch allows applications/APIs that access media through the core file
APIs to play media in the media cache. Prior to determining if a 'filename'
exists, the filename is passed to the media cache's retrieve API call. If
that call succeeds, the local file specified passed back by the API is
opened for streaming. When used in this fashion, the 'filename' is actually
a URI that the media cache process and understand.

ASTERISK-25654 #close

Change-Id: I73b6e2e90c3e91b8500581c45cdf9c0dc785f5f0

This patch adds unit tests for res_http_media cache, that covers nominal
creation and retrieval - and through them as well, staleness and deletion
checks. In addition, this patch adds tests that covers the interaction of
various HTTP headers, including Expires, Etag, and Cache-Control.

ASTERISK-25654

Change-Id: I2db101e307c863857fe416d6f5bf4cace9ac7cf5

This patch adds a bucket backend for the core media cache that interfaces to a
remote HTTP server. When a media item is requested in the cache, the cache will
query its bucket backends to see if they can provide the media item. If that
media item has a scheme of HTTP or HTTPS, this backend will be invoked.

The backend provides callbacks for the following:
 * create - this will always retrieve the URI specified by the provided
            bucket_file, and store it in the file specified by the object.
 * retrieve - this will pull the URI specified and store it in a temporary
              file. It is then up to the media cache to move/rename this file
              if desired.
 * delete - destroys the file associated with the bucket_file.
 * stale - if the bucket_file has expired, based on received HTTP headers from
           the remote server, or if the ETag on the server no longer matches
           the ETag stored on the bucket_file, the resource is determined to be
           stale.

Note that the backend respects the ETag, Expires, and Cache-Control headers
provided by the HTTP server it is querying.

ASTERISK-25654

Change-Id: Ie201c2b34cafc0c90a7ee18d7c8359afaccc5250

This patch does the following:

First, it addresses file extension handling in the media cache. The media core
in Asterisk is a bit interesting in that it wants:
 * A file to have an extension on it. That extension is used to associate the
   file with a defined format module.
 * The filename passed to the core to not have an extension on it. This allows
   the core to match the available file formats with the format a channel
   is capable of handling.

Unfortunately, this makes the current implementation a bit lacking in the media
cache. By default, we do not store the extension of a retrieved URI on the
local file that is created. As a result, the media core does not know what
format the file is, and the file is ignored. Modifying the file outside of the
media core is bad, as we would not be able to update the internal
ast_bucket_file's path.

At the same time, we do not want to pass the extension out in the file_path
parameter in ast_media_cache_retrieve. This parameter is intended to be fed
into the media core; if we passed the extension, all callers would have to
strip it off.

Thus, this patch does the following:
* If there is an extension specified in the URL, we append it to the local
  file name (if a preferred file name isn't specified), and we store that
  in the local file path.
* The extension, however, is stripped off of the file_path parameter passed
  back out of ast_media_cache_retrieve.

Second, this patch causes stale items to be completely removed from the system.
Prior to this patch, sound files could be orphaned due to the bucket
referencing the file being deleted, but the file itself not being removed. This
is now addressed by explicitly calling ast_bucket_file_delete on the
bucket_file when it is deemed to be stale. Note that this only happen when we
know we will attempt to retrieve the resource again.

Finally, this patch changes the AO2 container holding media items to just use
a regular mutex. The usage for this container already assumed it was a plain
mutex, and - given that retrieval of an item can cause it to be replaced in
the container - a mutex makes more sense than a read/write lock.

Change-Id: I51667fff86ae8d2e4a663555dfa85b11e935fe0f

This patch adds a write option to the CURL dialplan function, allowing it to
CURL files and store them locally. The value 'written' to the CURL URL
specifies the location on disk to store the file. As an example:

same => n,Set(CURL(http://1.1.1.1/foo.wav)=/tmp/foo.wav)

Would retrieve the file foo.wav from the remote server and store it in the
/tmp directory.

Due to the potentially dangerous nature of this function call, APIs are
forbidden from using the write functionality unless live_dangerously is set
to True in asterisk.conf.

ASTERISK-25652 #close

Change-Id: I44f4ad823d7d20f04ceaad3698c5c7f653c41b0d

PortAudio should no longer be required
PJSIP_MAX_PKT_LEN is now 6000
Older autoconf issue fixed. (CentOS 6)

Change-Id: I463fa9586cbe7c6b3b603289f535bd8e361611dd

check_via() already skips leading blanks where the sent-by address (with the
optional port) should be placed.

Since RFC 3261 allows for blanks between the port ant the Via parameters:
> https://tools.ietf.org/html/rfc3261#section-20.42
(actually it allows a lot of blanks more ;-)). I just switched from
ast_skip_blanks() to ast_strip() on the local copy of the string.

ASTERISK-21301 #close

Change-Id: Ie5b8fe5a07067b7c0dc9bcdd1707e99b23b02b06

We don't need pjproject's documentation embedded in Asterisk's.

Change-Id: Iea6f5a621c0f4e3168dda3321eaab258d9f24a17

The encryption code for AES_ENCRYPT evaluates the length of the data to
be encoded in base64 using strlen. The data is binary, thus the length
of it can be underestimated at the first NULL character.
Reuse the write pointer offset to evaluate it, instead.

ASTERISK-25857 #close

Change-Id: If686b5d570473eb926693c73461177b35b13b186

Fix the reference leak introduced in the following commit:

c534bd58

ASTERISK-25849

Change-Id: I5cfefd5ee6c1c3a1715c050330aaa10e4d2a5e85

During a transfer involving direct media a race occurs between when the
transferer channel is swapped out, initiating rtp changes/updates, and the
subsequent reinvites.

When Alice, after speaking with Charlie (Bob is on hold), connects Bob and
Charlie invites are sent to each in order to establish the call between them.
Bob is taken off hold and Charlie is told to have his media flow through
Asterisk. However, if before those invites go out the bridge updates Bob's
and/or Charlie's rtp information with direct media data (i.e. address, port)
then the invite(s) will contain the remote data in the SDP instead of the
Asterisk data.

The race occurs in the native bridge glue code when updating the peer. The
direct_media_address can get set twice before sending out the first invite
during call connection. This can happen because the checking/setting of the
direct_media_address happened in one thread while the sending of the invite(s)
happened in another thread.

This fix removes the race condition by moving the checking/setting of the
direct_media_address to be in the same thread as the sending of the invites(s).
This serializes the checking/setting and sending so they can no longer happen
out of order.

ASTERISK-25849 #close

Change-Id: Idfea590175e74f401929a601dba0c91ca1a7f873

The "Refer-to" header of an incoming REFER request is parsed by
pjsip_parse_uri().  That function requires the URI parameter to be NULL
terminated.  Unfortunately, the previous code added the NULL terminator by
overwriting memory that may not be safe.  The overwritten memory results
could be benign, memory corruption, or a segmentation fault.  Now the URI
is NULL terminated safely by copying the URI to a new chunk of memory with
the correct size to be NULL terminated.

ASTERISK-25814 #close

Change-Id: I32565496684a5a49c3278fce06474b8c94b37342