The file playback system will now query the media cache and then
the old file functionality. Under normal conditions this will result
in the cache failing to retrieve a file causing a warning message
to get output each time a file is played back.

This change demotes this warning to a debug message.

Change-Id: Ib72246ba300b5cce32774bfb3c26634bfb708624

There is a little known feature in app_controlplayback that will cause the
specified offset to be used relative to the end of a file if a ':end' is
detected within the filename.

This feature is pretty bad, but okay.

However, a bug exists in this code where a ':' detected in the filename
will cause the end pointer to be non-NULL, even if the full ':end' isn't
specified. This causes us to treat an unspecified offset (0) as being
"start playing from the end of the file", resulting in no file playback
occurring.

This patch fixes this bug by resetting the end pointer if ':end' is not
found in the filename.

Change-Id: Ib4c7b1b45283e4effd622a970055c51146892f35

This patch allows applications/APIs that access media through the core file
APIs to play media in the media cache. Prior to determining if a 'filename'
exists, the filename is passed to the media cache's retrieve API call. If
that call succeeds, the local file specified passed back by the API is
opened for streaming. When used in this fashion, the 'filename' is actually
a URI that the media cache process and understand.

ASTERISK-25654 #close

Change-Id: I73b6e2e90c3e91b8500581c45cdf9c0dc785f5f0

This patch does the following:

First, it addresses file extension handling in the media cache. The media core
in Asterisk is a bit interesting in that it wants:
 * A file to have an extension on it. That extension is used to associate the
   file with a defined format module.
 * The filename passed to the core to not have an extension on it. This allows
   the core to match the available file formats with the format a channel
   is capable of handling.

Unfortunately, this makes the current implementation a bit lacking in the media
cache. By default, we do not store the extension of a retrieved URI on the
local file that is created. As a result, the media core does not know what
format the file is, and the file is ignored. Modifying the file outside of the
media core is bad, as we would not be able to update the internal
ast_bucket_file's path.

At the same time, we do not want to pass the extension out in the file_path
parameter in ast_media_cache_retrieve. This parameter is intended to be fed
into the media core; if we passed the extension, all callers would have to
strip it off.

Thus, this patch does the following:
* If there is an extension specified in the URL, we append it to the local
  file name (if a preferred file name isn't specified), and we store that
  in the local file path.
* The extension, however, is stripped off of the file_path parameter passed
  back out of ast_media_cache_retrieve.

Second, this patch causes stale items to be completely removed from the system.
Prior to this patch, sound files could be orphaned due to the bucket
referencing the file being deleted, but the file itself not being removed. This
is now addressed by explicitly calling ast_bucket_file_delete on the
bucket_file when it is deemed to be stale. Note that this only happen when we
know we will attempt to retrieve the resource again.

Finally, this patch changes the AO2 container holding media items to just use
a regular mutex. The usage for this container already assumed it was a plain
mutex, and - given that retrieval of an item can cause it to be replaced in
the container - a mutex makes more sense than a read/write lock.

Change-Id: I51667fff86ae8d2e4a663555dfa85b11e935fe0f

This patch is part of a series to resolve deadlocks in chan_sip.c.

* Updated sched unit test to check new behavior.

ASTERISK-25023

Change-Id: Ib69437327b3cda5e14c4238d9ff91b2531b34ef3

Channel masquerading had a conflict with autochannel locking.

When locking autochannel->channel, the channel is fetched from the
autochannel and then locked. During the fetch, the autochannel -- which
has no locks itself -- can be modified by someone who owns the channel
lock. That means that the value of autochan->channel cannot be trusted
until you hold the lock.

In practice, this caused problems with Local channels getting
masqueraded away while the ChanSpy attempted to get info from that
channel. The old channel which was about to get removed got locked, but
the new (replaced) channel got unlocked (no-op). Because the replaced
channel was now locked (and would never get unlocked), it couldn't get
removed from the channel list in a timely manner, and would now cause
deadlocks when iterating over the channel list.

This change checks the autochannel after locking the channel for changes
to the autochannel. If the channel had been changed, the lock is
reobtained on the new channel.

In theory it seems possible that after this fix, the lock attempt on the
old (wrong) channel can be on an already destroyed lock, maybe causing
a crash. But that hasn't been observed in the wild and is harder induce
than the current deadlock.

Thanks go to Filip Frank for suggesting a fix similar to this and
especially to IRC user hexanol for pointing out why this deadlock was
possible and testing this fix. And to Richard for catching my rookie
while loop mistake ;)

ASTERISK-25321 #close

Change-Id: I293ae0014e531cd0e675c3f02d1d118a98683def

Refactor and created function ast_cli_print_timestr_fromseconds to print
seconds formatted:  year(s) week(s) day(s) hour(s) second(s)

This function now is used in addons/cdr_mysql.c,cdr_pgsql.c, main/cli.c,
res_config_ldap.c, res_config_pgsql.c.

Change-Id: Ibeb8634102cd11d3f8623398b279cb731bcde36c

Although we use the RTLD_LAZY flag when calling dlopen
the first time on a module, this only defers resolution
for function calls.  Pointer references to functions are
determined at link time so dlopen expects them to be there.
Since we don't cross-module link, pointers to functions
in other modules won't be available and dlopen will fail.

Doing a "hardened" build also causes problems because it
typically sets "-z now" on the ld command line which
overrides RTLD_LAZY at run time.

If the failing module isn't a GLOBAL_SYMBOLS module, then
dlopen will be called again after all the GLOBAL_SYMBOLS
modules have been loaded and they'll eventually resolve.

If the calling module IS a GLOBAL_SYMBOLS module itself
and a third module depends on it, then there's an issue
because the second time through the dlopen loop,
GLOBAL_SYMBOLS modules aren't given any special treatment
and since the order in which dlopen is called isn't
deterministic, the dependent may again be tried before the
module it needs is loaded.

Simple solution:  Save modules that fail load_resource
because of a dlopen error in a list and retry them
immediately after the first pass. Keep retrying until
the failed list is empty or we reach a #defined max
retries. Error messages are suppressed until the final
pass which also gets rid of those confusing error messages
about module failures that are later corrected.

Change-Id: Iddae1d97cd2f00b94e61662447432765755f64bb

It's possible for the transferer channel to get hung up early during the
attended transfer process. For instance, a phone may send a "bye" immediately
upon receiving a sip notify that contains a sip frag 100 (I'm looking at you
Jitsi). When this occurs a race begins between the transferer being hung up
and completion of the transfer code.

If the channel hangs up too early during a transfer involving stasis bridging
for instance, then when the created local channel goes to look up its swap
channel (and associated datastore) it can't find it (since it is no longer in
the bridge) thus it fails to enter the stasis application. Consequently, the
created local channel(s) hang up as well. If the timing is just right then the
bridging code attempts to add the message link with missing local channel(s).
Hence the crash.

Unfortunately, there is no great way to solve the problem of the unexpected
"bye". While we can't guarantee we won't receive an early hangup, and in this
case still fail to enter the stasis application, we can make it so asterisk
does not crash.

This patch does just that by locking the local channel structure, checking
that the local channel's peer has not been lost, and then continuing. This
keeps the local channel's peer from being ripped out from underneath it by
the local/unreal hangup code while attempting to set the stasis message link.

ASTERISK-25771

Change-Id: Ie6d6061e34c7c95f07116fffac9a09e5d225c880

In message.c, if msg_alloc fails to init the string field,
vars may be null, so use a null tolerant cleanup.

In res_pjsip_messaging.c, if msg_data_create fails, mdata
will be null, so use a null tolerant cleanup.

ASTERISK-25323

Change-Id: Ic2d55c2c3750d5616e2a05ea92a19c717507ff56

Previous chan_sip behavior:

Before this patch chan_sip would always strip any quotes from an incoming
reason and pass that value up as the REDIRECTING(reason).  For an outgoing
reason value, chan_sip would check the value against known values and
quote any it didn't recognize.  Incoming 480 response message reason text
was just assigned to the REDIRECTING(reason).

Previous chan_pjsip behavior:

Before this patch chan_pjsip would always pass the incoming reason value
up as the REDIRECTING(reason).  For an outgoing reason value, chan_pjsip
would send the reason value as passed down.

With this patch:

Both channel drivers match incoming reason values with values documented
by REDIRECTING(reason) and values documented by RFC5806 regardless of
whether they are quoted or not.  RFC5806 values are mapped to the
equivalent REDIRECTING(reason) documented value and is set in
REDIRECTING(reason).  e.g., an incoming RFC5806 'unconditional' value or a
quoted string version ('"unconditional"') is converted to
REDIRECTING(reason)'s 'cfu' value.  The user's dialplan only needs to deal
with 'cfu' instead of any of the aliases.

The incoming 480 response reason text supported by chan_sip checks for
known reason values and if not matched then puts quotes around the reason
string and assigns that to REDIRECTING(reason).

Both channel drivers send outgoing known REDIRECTING(reason) values as the
unquoted RFC5806 equivalent.  User custom values are either sent as is or
with added quotes if SIP doesn't allow a character within the value as
part of a RFC3261 Section 25.1 token.  Note that there are still
limitations on what characters can be put in a custom user value.  e.g.,
embedding quotes in the middle of the reason string is silly and just
going to cause you grief.

* Setting a REDIRECTING(reason) value now recognizes RFC5806 aliases.
e.g., Setting REDIRECTING(reason) to 'unconditional' is converted to the
'cfu' value.

* Added missing malloc() NULL return check in res_pjsip_diversion.c
set_redirecting_reason().

* Fixed potential read from a stale pointer in res_pjsip_diversion.c
add_diversion_header().  The reason string needed to be copied into the
tdata memory pool to ensure that the string would always be available.
Otherwise, if the reason string returned by reason_code_to_str() was a
user's reason string then the string could be freed later by another
thread.

Change-Id: Ifba83d23a195a9f64d55b9c681d2e62476b68a87

Background here:
http://lists.digium.com/pipermail/asterisk-dev/2016-January/075266.html

From CHANGES:
 * To help insure that Asterisk is compiled and run with the same known
   version of pjproject, a new option (--with-pjproject-bundled) has been
   added to ./configure.  When specified, the version of pjproject specified
   in third-party/versions.mak will be downloaded and configured.  When you
   make Asterisk, the build process will also automatically build pjproject
   and Asterisk will be statically linked to it.  Once a particular version
   of pjproject is configured and built, it won't be configured or built
   again unless you run a 'make distclean'.

   To facilitate testing, when 'make install' is run, the pjsua and pjsystest
   utilities and the pjproject python bindings will be installed in
   ASTDATADIR/third-party/pjproject.

   The default behavior remains building with the shared pjproject
   installation, if any.

Building:

   All you have to do is include the --with-pjproject-bundled option on
   the ./configure command line (and remove any existing --with-pjproject
   option if specified).  Everything else is automatic.

Behind the scenes:

   The top-level Makefile was modified to include 'third-party' in the
   list of MOD_SUBDIRS.

   The third-party directory was created to contain any third party
   packages that may be needed in the future.  Its Makefile automatically
   iterates over any subdirectories passing on targets.

   The third-party/pjproject directory was created to house the pjproject
   source distribution.  Its Makefile contains targets to download, patch
   configure, generate dependencies, compile libs, apps and python bindings,
   sanitized build.mak and generate a symbols list.

   When bootstrap.sh is run, it automatically includes the configure.m4
   file in third-party/pjproject.  This file has a macro to download and
   conifgure pjproject and get and set PJPROJECT_INCLUDE, PJPROJECT_DIR
   and PJPROJECT_BUNDLED.  It also tests for the capabilities like
   PJ_TRANSACTION_GRP_LOCK by parsing preprocessor output as opposed to
   trying to compile.  Of course, bootstrap.sh is only run once and the
   configure file is incldued in the patch.

   When configure is run with the new options, the macro in configure.m4
   triggers the download, patch, conifgure and tests.  No compilation is
   performed at this time.  The downloaded tarball is cached in /tmp so
   it doesn't get downloaded again on a distclean.

   When make is run in the top-level Asterisk source directory, it will
   automatically descend all the subdirectories in third_party just as it
   does for addons, apps, etc.  The top-level Makefile makes sure that
   the 'third-party' is built before 'main' so that dependencies from the
   other directories are built first.

   When main does build, a new shared library (libasteriskpj) is created that
   links statically to the pjproject .a files and exports all their symbols.
   The asterisk binary links to that, just as it does with libasteriskssl.

   When Asterisk is installed, the pjsua and pjsystest apps, and the pjproject
   python bindings are installed in ASTDATADIR/third-party/pjproject.  This
   will facilitate testing, including running the testsuite which will be
   updated to check that directory for the pjsua module ahead of the system
   python library.

Modules should continue to depend on pjproject if they use pjproject APIs
directly.  They should not care about the implementation.  No changes to any
res_pjsip modules were made.

Change-Id: Ia7a60c28c2e9ba9537c5570f933c1ebcb20a3103

The channel is now going to get T.38 terminated when it leaves the
bridging system and the bridged peers are going to get T.38 terminated as
well.

ASTERISK-25582

Change-Id: I77a9205979910210e3068e1ddff400dbf35c4ca7

ASTERISK-25582

Change-Id: I69451920b122de7ee18d15bb231c80ea7067a22b

Local channel optimization could cause DTMF digits to be duplicated.
Pending DTMF end events would be posted to a bridge when the local channel
optimizes out and is replaced by the channel further down the chain.  When
the real digit ends, the channel would get another DTMF end posted to the
bridge.

A -- LocalA;1/n -- LocalA;2/n -- LocalB;1 -- LocalB;2 -- B

1) LocalA has the /n flag to prevent optimization.
2) B is sending DTMF to A through the local channel chain.
3) When LocalB optimizes out it can move B to the position of LocalB;1
4) Without this patch, when B swaps with LocalB;1 then LocalB;1 would
settle an owed DTMF end to the bridge toward LocalA;2.
5) When B finally ends its DTMF it sends the DTMF end down the chain.
6) Without this patch, A would hear the DTMF digit end when LocalB
optimizes out and when B ends the original digit.

ASTERISK-25582

Change-Id: I1bbd28b8b399c0fb54985a5747f330a4cd2aa251

Frame hooks can conceivably return a control frame in exchange for an
audio frame inside ast_write().  Those returned control frames were not
handled quite the same as if they were sent to ast_indicate().  Now it
doesn't matter if you use ast_write() to send an AST_FRAME_CONTROL to a
channel or ast_indicate().

ASTERISK-25582

Change-Id: I5775f41421aca2b510128198e9b827bf9169629b

The ast_sorcery_create, update and delete function have been refactored
to better deal with caches and errors.

The action is now called on all non-caching wizards first. If ANY succeed,
the action is called on all caching wizards and the observers are notified.
This way we don't put something in the cache (or update or delete) before
knowing the action was performed in at least 1 backend and we only call the
observers once even if there were multiple writable backends.

ast_sorcery_create was never adding to caches in the first place which
was preventing contacts from getting added to a memory_cache when they
were created.  In turn this was causing memory_cache to emit errors if
the contact was deleted before being retrieved (which would have
populated the cache).

ASTERISK-25811 #close
Reported-by: Ross Beer

Change-Id: Id5596ce691685a79886e57b0865888458d6e7b46

The return type of ast_cel_track_event() is not large enough to return all
64 potential bits of the event enable mask.  Fortunately, the defined CEL
events do not really need all 64 bits and the return value is only used to
determine if the requested CEL event is enabled.

* Made the ast_cel_track_event() return 0 or 1 only so the return value
can fit inside an int type instead of zero or a truncated 64 bit non-zero
value.

Change-Id: I783d932320db11a95c7bf7636a72b6fe2566904c

The SSL_OP_NO_TLSv1_1 and SSL_OP_NO_TLSv1_2 defines did not exist prior
to OpenSSL version 1.0.1. A recent commit attempts to, by default, set
these options, which can cause problems on systems with older OpenSSL
installations.

This commit adds a configure script check for those defines and will not
attempt to make use of those if they do not exist. We will print a
warning urging the user to upgrade their OpenSSL installation if those
defines are not present.

Change-Id: I6a2eb9a43fd0738b404d8f6f2cf4b5c22d9d752d

This change exposes the configuration of various aspects of the TLS
support and sets the default to the modern standards.

The TLS cipher is now set to the best values according to the
Mozilla OpSec team, different TLS versions can now be disabled, and
the cipher order can be forced to be that of the server instead of
the client.

ASTERISK-24972 #close

Change-Id: I0a10f2883f7559af5e48dee0901251dbf30d45b8

Sending UDPTL packets to Asterisk with the right amount of missing
sequence numbers and enough redundant 0-length IFP packets, can make
Asterisk crash.

ASTERISK-25603 #close
Reported by: Walter Doekes

ASTERISK-25742 #close
Reported by: Torrey Searle

Change-Id: I97df8375041be986f3f266ac1946a538023a5255

Fix some warnings found with clang.

Change-Id: I5195b6189b148c2ee3ed4a19d015a6d4ef3e77bd

Change-Id: I915ea437936320393afde0e7552cf0a980a6b2e4

The config options framework is strict in that configuration options must
be documented unless XML documentation support is not available. In
practice this is useful as it ensures documentation exists however in
off-nominal cases this can cause strange problems.

If it is expected that a config option has a non-zero or non-empty
default value but the config option documentation is unavailable
this reasonable expectation will not be met. This can cause obscure
crashes and weirdness depending on how the code handles it.

This change tweaks the behavior to ensure that the config option
is still allowed to register, apply default values, and be set when
devmode is not enabled. If devmode is enabled then the option can
NOT be set.

This also does not remove the initial documentation error message that
is output on load when registering the configuration option.

ASTERISK-25725 #close

Change-Id: Iec42fca6b35f31326c33fcdc25473f6fd7bc8af8

The null terminator of the tail struct member was not being allocated
when no logger.conf config file is installed.

ASTERISK-25714 #close
Reported by: Badalian Vyacheslav

Change-Id: I45770fdd08af39506a3bc33ba279c4f16e047a30

Make sure buf[res] is not accessed at res=-1 (buffer underrun).
Address Sanitizer will complain about this quite loudly.

ASTERISK-24801 #close

Change-Id: Ifcd7f691310815a31756b76067c56fba299d3ae9

Change-Id: I4892d6acbb580d6c207d006341eaf5e0f8f2a029

You have to call ast_taskprocessor_unref() outside of the taskprocessor
implementation code.  Taskprocessor use since v12 has become more
transient than just the singleton uses in earlier versions.

Change-Id: If7675299924c0cc65f2a43a85254e6f06f2d61bb

This removes logchannels locking from init_logger_chain, puts the
responsibility on the caller.  Adds locking around the one call that was
missing it.

ASTERISK-24833

Change-Id: I6cc42117338bf9575650a67bcb78ab1a33d7bad8

ASTERISK-25700 #close

Change-Id: I096da84f9c62c6095f68bcf98eac4b7c7868e808

The xferfailsound was read from the channel at the beginning of the transfer,
and that value is "cached" for the duration of the transfer. Therefore, changing
the xferfailsound on the channel using the FEATURE() dialplan function does
nothing once the transfer is under way.

This makes it so the transfer code instead gets the xferfailsound configuration
options from the channel when it is actually going to be used.

This patch also fixes a potential memory leak of the props object as well as
making sure the condition variable gets initialized before being destroyed.

ASTERISK-25696 #close

Change-Id: Ic726b0f54ef588bd9c9c67f4b0e4d787934f85e4

Change-Id: Id5bd18ef1f60ef8be453e677e98478298358a9d1

* Add freed regions totals to allocations and summary.

* Add totals for all allocations and not just the selected allocations.

Change-Id: I61d5a5112617b0733097f2545a3006a344b4032a

If the attended transfer destination answers (picks call up or goes to
voicemail) and then hangs up on the transferer then transferer hears the
fail sound.

This patch makes it so the fail sound is not played when the transfer
destination/target hangs up after answering.

ASTERISK-25697 #close

Change-Id: I97f142fe4fc2805d1a24b7c16143069dc03d9ded

This issue was exposed when executing a connected line subroutine.
When connected or redirected subroutines or macros are executed it is
expected that the underlying applications and logic invoked are fast
and do not consume frames. In practice this constraint is not enforced
and if not adhered to will cause channels to continue when they shouldn't.
This is because each caller of the connected or redirected logic does not
check whether the channel has been hung up on return. As a result the
the hung up channel continues.

This change makes it so when the API to execute a subroutine or
macro is invoked the channel is checked to determine if it has hung up.
If it has then a hangup is queued again so the caller will see it
and stop.

ASTERISK-25690 #close

Change-Id: I1f9a8ceb1487df0389f0d346ce0f6dcbcaf476ea

Recent changes (ASTERISK-25394 commit 2bd27d12)
introduced the possibility of a deadlock. Due to the mentioned modifications
ast_change_hints now needs to keep both merge/delete and state callbacks from
occurring while it executes. Unfortunately, sometimes ast_change_hints can be
called with the contexts container locked. When this happens it's possible for
another thread to grab the context_merge_lock before the thread calling into
ast_change_hints does and then try to obtain the contexts container lock. This
of course causes a deadlock between the two threads. The thread calling into
ast_change_hints waits for the other thread to release context_merge_lock and
the other thread is waiting on that one to release the contexts container lock.

Unfortunately, there is not a great way to fix this problem. When hints change,
the subsequent state callbacks cannot run at the same time as a merge/delete,
nor when the usual state callbacks do. This patch alleviates the problem by
having those particular callbacks (the ones run after a hint change) occur in a
serialized task. By moving the context_merge_lock to a task it can now safely be
attempted or held without a deadlock occurring.

ASTERISK-25640 #close
Reported by: Krzysztof Trempala

Change-Id: If2210ea241afd1585dc2594c16faff84579bf302

ASTERISK-25681 #close

Change-Id: I64337c70f0ebd8c77f70792042684607c950c8f1

ASTERISK-25680 #close

Change-Id: I3251d781cbc3f48a6a7e1b969ac4983f552b2446

Sorcery name formats:
sorcery/<type>-<seq> -- Sorcery thread pool serializer

Change-Id: Idc2e5d3dbab15c825b97c38c028319a0d2315c47

Stasis name formats:
subm:<topic>-<seq> -- Stasis subscription mailbox task processor
subp:<topic>-<seq> -- Stasis subscription thread pool serializer

Change-Id: Id19234b306e3594530bb040bc95d977f18ac7bfd