ASTERISK-29756

Change-Id: I2f1c1eea1c902492b77b74de9950f20ebbb7e758

This change ensures that the module isn't unloaded when a
WebSocket is open. Previously it was possible to unload the
module manually or during shutdown which could cause a crash
when any active WebSockets were terminated.

ASTERISK-28585

Change-Id: I85c71ab112f99875b586419a34c08c8b34c14c5c

This removes references that are no longer needed due to automatic
references created by module dependencies.

In addition this removes most calls to ast_module_check as they were
checking modules which are listed as dependencies.

Change-Id: I332a6e8383d4c72c8e89d988a184ab8320c4872e

I've audited all modules that include any header which includes
asterisk/optional_api.h.  All modules which use OPTIONAL_API now declare
those dependencies in AST_MODULE_INFO using requires or optional_modules
as appropriate.

In addition ARI dependency declarations have been reworked.  Instead of
declaring additional required modules in res/ari/resource_*.c we now add
them to an optional array "requiresModules" in api-docs for each module.
This allows the AST_MODULE_INFO dependencies to include those missing
modules.

Change-Id: Ia0c70571f5566784f63605e78e1ceccb4f79c606

* Declare 'requires' and 'enhances' text fields on module info structure.
* Rename 'nonoptreq' to 'optional_modules'.
* Update doxygen comments.

Still need to investigate dependencies among modules I cannot compile.

Change-Id: I3ad9547a0a6442409ff4e352a6d897bef2cc04bf

The recent change to make the use of LOAD_DECLINE more consistent
caused res_ari to unload itself before declining if the ari.conf
file wasn't found.  The ari stubs though still tried to use the
configuration resulting in segfaults.

This patch creates a new CHECK_ARI_MODULE_LOADED macro which tests
to see if res_ari is actually loaded and causes the stubs to also
decline if it isn't.  The macro was then added to the mustache
template's "load_module" function.

ASTERISK-27026 #close
Reported-by: Ronald Raikes

Change-Id: I263d56efa628ee3c411bdcd16d49af6260c6c91d

The periodic doc job does a make ari-stubs and checks that
there are no changes before generating the docs.  Since I changed
the mustache template (and the generated code directly) recently
and forgot to regenerate the stubs, the doc job thinks they're out
of date.

Change-Id: I94b97035311eccf52b0101b8590223265a7881d4

Change-Id: If99e3b4fc2d7e86fc3e61182aa6c835b407ed49e

In all non-pbx modules, AST_MODULE_LOAD_FAILURE has been changed
to AST_MODULE_LOAD_DECLINE.  This prevents asterisk from exiting
if a module can't be loaded.  If the user wishes to retain the
FAILURE behavior for a specific module, they can use the "require"
or "preload-require" keyword in modules.conf.

A new API was added to logger: ast_is_logger_initialized().  This
allows asterisk.c/check_init() to print to the error log once the
logger subsystem is ready instead of just to stdout.  If something
does fail before the logger is initialized, we now print to stderr
instead of stdout.

Change-Id: I5f4b50623d9b5a6cb7c5624a8c5c1274c13b2b25

The 'ari set debug' command has been enhanced to accept 'all' as an
application name.  This allows dumping of all apps even if an app
hasn't registered yet.  To accomplish this, a new global_debug global
variable was added to res/stasis/app.c and new APIs were added to
set and query the value.

'ari set debug' now displays requests and responses as well as events.
This required refactoring the existing debug code.

* The implementation for 'ari set debug' was moved from stasis/cli.{c,h}
  to ari/cli.{c,h}, and stasis/cli.{c,h} were deleted.
* In order to print the body of incoming requests even if a request
  failed, the consumption of the body was moved from the ari stubs
  to ast_ari_callback in res_ari.c and the moustache templates were
  then regenerated.  The body is now passed to ast_ari_invoke and then
  on to the handlers.  This results in code savings since that template
  was inserted multiple times into all the stubs.

An additional change was made to the ao2_str_container implementation
to add partial key searching and a sort function.  The existing cli
code assumed it was already there when it wasn't so the tab completion
was never working.

Change-Id: Ief936f747ce47f1fb14035fbe61152cf766406bf
(cherry picked from commit 1d890874)

ASTERISK_REGISTER_FILE no longer has any purpose so this commit removes
all traces of it.

Previously exported symbols removed:
* __ast_register_file
* __ast_unregister_file
* ast_complete_source_filename

This also removes the mtx_prof static variable that was declared when
MTX_PROFILE was enabled.  This variable was only used in lock.c so it
is now initialized in that file only.

ASTERISK-26480 #close

Change-Id: I1074af07d71f9e159c48ef36631aa432c86f9966

Given resource paths did not have 'json' substituted in for the '{format}'. For
some auto generated documentation/comment strings it resulted in something like
the following:

"... REST handler for /api-docs/sounds.{format}"

This patch makes sure the resource api's path is properly substituted.

ASTERISK-25472 #close

Change-Id: Ie3e950a35db4043e284019d6c9061f3b03922e23

This patch adds the ability to subscribe to all events. There are two possible
ways to accomplish this:
(1) On initial WebSocket connection. This patch adds a new query parameter,
    'subscribeAll'. If present and True, Asterisk will subscribe the
    applications to all ARI events.
(2) Via the applications resource. When subscribing in this manner, an ARI
    client should merely specify a blank resource name, i.e., 'channels:'
    instead of 'channels:12354'. This will subscribe the application to all
    resources of the 'channels' type.

ASTERISK-24870 #close

Change-Id: I4a943b4db24442cf28bc64b24bfd541249790ad6

ASTERISK-25308 #close
Reported by: Joshua Colp

Change-Id: I592785bf70ff4b63d00e535b482f40da8e82a082

Prior to ASTERISK-24988, the WebSocket handshake was resolved before Stasis
applications were registered. This was done such that the WebSocket would be
ready when an application is registered. However, by creating the WebSocket
first, the client had the ability to make requests for the Stasis application
it thought had been created with the initial handshake request. The inevitable
conclusion of this scenario was the cart being put before the horse.

ASTERISK-24988 resolved half of the problem by ensuring that the applications
were created and registered with Stasis prior to completing the handshake
with the client. While this meant that Stasis was ready when the client
received the green-light from Asterisk, it also meant that the WebSocket was
not yet ready for Stasis to dispatch messages.

This patch introduces a message queuing mechanism for delaying messages from
Stasis applications while the WebSocket is being constructed. When the ARI
event processor receives the message from the WebSocket that it is being
created, the event processor instantiates an event session which contains a
message queue. It then tries to create and register the requested applications
with Stasis. Messages that are dispatched from Stasis between this point and
the point at which the event processor is notified the WebSocket is ready, are
stashed in the queue. Once the WebSocket has been built, the queue's messages
are dispatched in the order in which they were originally received and the
queue is concurrently cleared.

ASTERISK-25181 #close
Reported By: Matt Jordan

Change-Id: Iafef7b85a2e0bf78c114db4c87ffc3d16d671a17

Prior to this patch, when a WebSocket connection is made, ARI would not
be informed of the connection until after the WebSocket layer had
accepted the connection. This created a brief race condition where the
ARI client would be notified that it was connected, a channel would be
sent into the Stasis dialplan application, but ARI would not yet have
registered the Stasis application presented in the HTTP request that
established the WebSocket.

This patch resolves this issue by doing the following:
 * When a WebSocket attempt is made, a callback is made into the ARI
   application layer, which verifies and registers the apps presented in
   the HTTP request. Because we do not yet have a WebSocket, we cannot
   have an event session for the corresponding applications. Some
   defensive checks were thus added to make the application objects
   tolerant to a NULL event session.
 * When a WebSocket connection is made, the registered application is
   updated with the newly created event session that wraps the WebSocket
   connection.

ASTERISK-24988 #close
Reported by: Joshua Colp

Change-Id: Ia5dc60dc2b6bee76cd5aff0f69dd53b36e83f636

Change-Id: Icf88f9f861c6b2a16e5f626ff25795218a6f2723

ARI modules that are generated by 'make ari-stubs' are all dependent on
res_ari_model.  Additionally some of the same modules depend on one or more
res_stasis_* modules.

ASTERISK-25027 #close
Reported by: Corey Farrell

Change-Id: I8e07fe7e81fedacb87232f2b6f8b5f47927b4153

Git does not support the ability to replace a token with a version
string during check-in. While it does have support for replacing a
token on clone, this is somewhat sub-optimal: the token is replaced
with the object hash, which is not particularly easy for human
consumption. What's more, in practice, the source file version was often
not terribly useful. Generally, when triaging bugs, the overall version
of Asterisk is far more useful than an individual SVN version of a file. As a
result, this patch removes Asterisk's support for showing source file
versions.

Specifically, it does the following:

* Rename ASTERISK_FILE_VERSION macro to ASTERISK_REGISTER_FILE, and
  remove passing the version in with the macro. Other facilities
  than 'core show file version' make use of the file names, such as
  setting a debug level only on a specific file. As such, the act of
  registering source files with the Asterisk core still has use. The
  macro rename now reflects the new macro purpose.

* main/asterisk:
  - Refactor the file_version structure to reflect that it no longer
    tracks a version field.
  - Remove the "core show file version" CLI command. Without the file
    version, it is no longer useful.
  - Remove the ast_file_version_find function. The file version is no
    longer tracked.
  - Rename ast_register_file_version/ast_unregister_file_version to
    ast_register_file/ast_unregister_file, respectively.

* main/manager: Remove value from the Version key of the ModuleCheck
  Action. The actual key itself has not been removed, as doing so would
  absolutely constitute a backwards incompatible change. However, since
  the file version is no longer tracked, there is no need to attempt to
  include it in the Version key.

* UPGRADE: Add notes for:
  - Modification to the ModuleCheck AMI Action
  - Removal of the "core show file version" CLI command

Change-Id: I6cf0ff280e1668bf4957dc21f32a5ff43444a40e

Fixed memory leaks that were found in Asterisk.

ASTERISK-24693 #close
Reported by:  Kevin Harwell
Review: https://reviewboard.asterisk.org/r/4347/
........

Merged revisions 430999 from http://svn.asterisk.org/svn/asterisk/branches/13


git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@431010 65c4cc65-6c06-0410-ace0-fbb531ad65f3

ASTERISK-23919 #close
Reported by Malcolm Davenport

Review: https://reviewboard.asterisk.org/r/3802



git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@419592 65c4cc65-6c06-0410-ace0-fbb531ad65f3

User events can now be generated from ARI.  Events can be signalled with
arbitrary json variables, and include one or more of channel, bridge, or
endpoint snapshots.  An application must be specified which will receive
the event message (other applications can subscribe to it).  The message
will also be delivered via AMI provided a channel is attached.  Dialplan
generated user event messages are still transmitted via the channel, and
will only be received by a stasis application they are attached to or if
the channel is subscribed to.

This change also introduces the multi object blob mechanism used to send
multiple snapshot types in a single message.  The dialplan app UserEvent
was also changed to use multi object blob, and a new stasis message type
created to handle them.

ASTERISK-22697 #close
Review: https://reviewboard.asterisk.org/r/3494/
........

Merged revisions 414405 from http://svn.asterisk.org/svn/asterisk/branches/12


git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@414406 65c4cc65-6c06-0410-ace0-fbb531ad65f3

While working on building client libraries from the Swagger API, I
noticed a problem with the nicknames.

    channel.deleteChannel()
    channel.answerChannel()
    channel.muteChannel()

Etc. We put the object name in the nickname (since we were generating C
code), but it makes OO generators redundant.

This patch makes the nicknames more OO friendly. This resulted in a lot
of name changing within the res_ari_*.so modules, but not much else.

There were a couple of other fixed I made in the process.

 * When reversible operations (POST /hold, POST /unhold) were made more
   RESTful (POST /hold, DELETE /unhold), the path for the second operation
   was left in the API declaration. This worked, but really the two
   operations should have been on the same API.
 * The POST /unmute operation had still not been REST-ified.

Review: https://reviewboard.asterisk.org/r/2940/
........

Merged revisions 402528 from http://svn.asterisk.org/svn/asterisk/branches/12


git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@402529 65c4cc65-6c06-0410-ace0-fbb531ad65f3

With the new work in Asterisk 12, there are some uses of the
optional_api that are prone to failure. The details are rather involved,
and captured on [the wiki][1].

This patch addresses the issue by removing almost all of the magic from
the optional API implementation. Instead of relying on weak symbol
resolution, a new optional_api.c module was added to Asterisk core.

For modules providing an optional API, the pointer to the implementation
function is registered with the core. For modules that use an optional
API, a pointer to a stub function, along with a optional_ref function
pointer are registered with the core. The optional_ref function pointers
is set to the implementation function when it's provided, or the stub
function when it's now.

Since the implementation no longer relies on magic, it is now supported
on all platforms. In the spirit of choice, an OPTIONAL_API flag was
added, so we can disable the optional_api if needed (maybe it's buggy on
some bizarre platform I haven't tested on)

The AST_OPTIONAL_API*() macros themselves remained unchanged, so
existing code could remain unchanged. But to help with debugging the
optional_api, the patch limits the #include of optional API's to just
the modules using the API. This also reduces resource waste maintaining
optional_ref pointers that aren't used.

Other changes made as a part of this patch:
 * The stubs for http_websocket that wrap system calls set errno to
   ENOSYS.

 * res_http_websocket now properly increments module use count.

 * In loader.c, the while() wrappers around dlclose() were removed. The
   while(!dlclose()) is actually an anti-pattern, which can lead to
   infinite loops if the module you're attempting to unload exports a
   symbol that was directly linked to.

 * The special handling of nonoptreq on systems without weak symbol
   support was removed, since we no longer rely on weak symbols for
   optional_api.

 [1]: https://wiki.asterisk.org/wiki/x/wACUAQ

(closes issue ASTERISK-22296)
Reported by: Matt Jordan
Review: https://reviewboard.asterisk.org/r/2797/
........

Merged revisions 397989 from http://svn.asterisk.org/svn/asterisk/branches/12


git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@397990 65c4cc65-6c06-0410-ace0-fbb531ad65f3

Stasis events (which get distributed over the ARI WebSocket) are created
by subscribing to the channel_all_cached and bridge_all_cached topics,
filtering out events for channels/bridges currently subscribed to.

There are two issues with that. First was a race condition, where
messages in-flight to the master subscribe-to-all-things topic would get
sent out, even though the events happened before the channel was put
into Stasis. Secondly, as the number of channels and bridges grow in the
system, the work spent filtering messages becomes excessive.

Since r395954, individual channels and bridges have caching topics, and
can be subscribed to individually. This patch takes advantage, so that
channels and bridges are subscribed to on demand, instead of filtering
the global topics.

The one case where filtering is still required is handling BridgeMerge
messages, which are published directly to the bridge_all topic.

Other than the change to how subscriptions work, this patch mostly just
moves code around. Most of the work generating JSON objects from
messages was moved to .to_json handlers on the message types. The
callback functions handling app subscriptions were moved from res_stasis
(b/c they were global to the model) to stasis/app.c (b/c they are local
to the app now).

(closes issue ASTERISK-21969)
Reported by: Matt Jordan
Review: https://reviewboard.asterisk.org/r/2754/
........

Merged revisions 397816 from http://svn.asterisk.org/svn/asterisk/branches/12


git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@397820 65c4cc65-6c06-0410-ace0-fbb531ad65f3

Swagger allows parameters to be specified as 'allowMultiple', meaning
that the parameter may be specified as a comma separated list of
values.

I had written some of the API docs using that, but promptly forgot
about implementing it. This patch finally fills in that gap.

The codegen template was updated to represent 'allowMultiple' fields
as array/size fields in the _args structs. It also parses the comma
separated list using ast_app_separate_args(), so quoted strings in the
argument will be handled properly.

Review: https://reviewboard.asterisk.org/r/2698/


git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@396122 65c4cc65-6c06-0410-ace0-fbb531ad65f3

This renames all files and API calls from several variants of
Stasis-HTTP to ARI including:
* Stasis-HTTP -> ARI
* STASIS_HTTP -> ARI
* stasis_http -> ari (ast_ari for global symbols, file names as well)
* stasis http -> ARI

Review: https://reviewboard.asterisk.org/r/2706/
(closes issue ASTERISK-22136)


git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@395603 65c4cc65-6c06-0410-ace0-fbb531ad65f3

git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@393749 65c4cc65-6c06-0410-ace0-fbb531ad65f3

The Asterisk strategy of loading modules with RTLD_LAZY to extract metadata
from the module works well enough, until you try to take the address of a
function.

If a module takes the address of a function, that function needs to be
resolved at load time. That kinda defeats RTLD_LAZY.

This patch adds some ari_validator_{id}_fn() wrapper functions for safely
getting the function pointer from a different module.


git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@393576 65c4cc65-6c06-0410-ace0-fbb531ad65f3

This patch started with the simple idea of changing the /events data
model to be more sane. The original model would send out events like:

    { "stasis_start": { "args": [], "channel": { ... } } }

The event discriminator was the field name instead of being a value in
the object, due to limitations in how Swagger 1.1 could model objects.
While technically sufficient in communicating event information, it was
really difficult to deal with in terms of client side JSON handling.

This patch takes advantage of a proposed extension[1] to Swagger which
allows type variance through the use of a discriminator field. This had
a domino effect that made this a surprisingly large patch.

 [1]: https://groups.google.com/d/msg/wordnik-api/EC3rGajE0os/ey_5dBI_jWcJ

In changing the models, I also had to change the swagger_model.py
processor so it can handle the type discriminator and subtyping. I took
that a big step forward, and using that information to generate an
ari_model module, which can validate a JSON object against the Swagger
model.

The REST and WebSocket generators were changed to take advantage of the
validators. If compiled with AST_DEVMODE enabled, JSON objects that
don't match their corresponding models will not be sent out. For REST
API calls, a 500 Internal Server response is sent. For WebSockets, the
invalid JSON message is replaced with an error message.

Since this took over about half of the job of the existing JSON
generators, and the .to_json virtual function on messages took over the
other half, I reluctantly removed the generators.

The validators turned up all sorts of errors and inconsistencies in our
data models, and the code. These were cleaned up, with checks in the
code generator avoid some of the consistency problems in the future.

 * The model for a channel snapshot was trimmed down to match the
   information sent via AMI. Many of the field being sent were not
   useful in the general case.
 * The model for a bridge snapshot was updated to be more consistent
   with the other ARI models.

Another impact of introducing subtyping was that the swagger-codegen
documentation generator was insufficient (at least until it catches up
with Swagger 1.2). I wanted it to be easier to generate docs for the API
anyways, so I ported the wiki pages to use the Asterisk Swagger
generator. In the process, I was able to clean up many of the model
links, which would occasionally give inconsistent results on the wiki. I
also added error responses to the wiki docs, making the wiki
documentation more complete.

Finally, since Stasis-HTTP will now be named Asterisk REST Interface
(ARI), any new functions and files I created carry the ari_ prefix. I
changed a few stasis_http references to ari where it was non-intrusive
and made sense.

(closes issue ASTERISK-21885)
Review: https://reviewboard.asterisk.org/r/2639/



git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@393529 65c4cc65-6c06-0410-ace0-fbb531ad65f3

This patch moves the RESTful URL's around to more appropriate
locations for release.

The /stasis URL's are moved to /ari, since Asterisk REST Interface was
a more appropriate name than Stasis-HTTP. (Most of the code still has
stasis_http references, but they will be cleaned up after there are no
more outstanding branches that would have merge conflicts with such a
change).

A larger change was moving the ARI events WebSocket off of the shared
/ws URL to its permanent home on /ari/events. The Swagger code
generator was extended to handle "upgrade: websocket" and
"websocketProtocol:" attributes on an operation.

The WebSocket module was modified to better handle WebSocket servers
that have a single registered protocol handler. If a client
connections does not specify the Sec-WebSocket-Protocol header, and
the server has a single protocol handler registered, the WebSocket
server will go ahead and accept the client for that subprotocol.

(closes issue ASTERISK-21857)
Review: https://reviewboard.asterisk.org/r/2621/



git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@393528 65c4cc65-6c06-0410-ace0-fbb531ad65f3

The menuselect parser is very simple.  It looks for AST_MODULE_INFO and
uses any quoted string on that line as the module summary display.


git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@392777 65c4cc65-6c06-0410-ace0-fbb531ad65f3

This moves the JSON event generators out of the Stasis-HTTP modules and
into standalone JSON-related counterparts so that Stasis-HTTP and
res_stasis can depend on them without creating dependency cycles. This
also provides a future location for Swagger Model validator functions
once the generators for that code are written.

Review: https://reviewboard.asterisk.org/r/2534/


git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@388668 65c4cc65-6c06-0410-ace0-fbb531ad65f3

I've noticed when doing a graceful shutdown that the res_stasis_http.so
module gets unloaded before the modules that use it, which causes some
asserts during their unload.

While r386928 was a quick hack to get it to not assert and die, this
patch increases the use counts on res_stasis.so and res_stasis_http.so
properly. It's a bigger change than I expected, hence the review instead
of just committing it.

Review: https://reviewboard.asterisk.org/r/2489/


git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@388350 65c4cc65-6c06-0410-ace0-fbb531ad65f3

This change adds a framework in res_stasis for handling events from
channel topics. JSON event generation and validation code is created
from event documentation in rest-api/api-docs/events.json to assist in
JSON event generation, ensure consistency, and ensure that accurate
documentation is available for ALL events that are received by
res_stasis applications.

The userevent application has been refactored along with the code that
handles userevent channel blob events to pass the headers as key/value
pairs in the JSON blob. As a side-effect, app_userevent now handles
duplicate keys by overwriting the previous value.

Review: https://reviewboard.asterisk.org/r/2428/


(closes issue ASTERISK-21180)
Patch-By: Kinsey Moore <kmoore@digium.com>


git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@388275 65c4cc65-6c06-0410-ace0-fbb531ad65f3

The API itself is documented using Swagger, a lightweight mechanism for
documenting RESTful API's using JSON. This allows us to use swagger-ui
to provide executable documentation for the API, generate client
bindings in different languages, and generate a lot of the boilerplate
code for implementing the RESTful bindings. The API docs live in the
rest-api/ directory.

The RESTful bindings are generated from the Swagger API docs using a set
of Mustache templates.  The code generator is written in Python, and
uses Pystache. Pystache has no dependencies, and be installed easily
using pip. Code generation code lives in rest-api-templates/.

The generated code reduces a lot of boilerplate when it comes to
handling HTTP requests. It also helps us have greater consistency in the
REST API.

(closes issue ASTERISK-20891)
Review: https://reviewboard.asterisk.org/r/2376/

git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@386232 65c4cc65-6c06-0410-ace0-fbb531ad65f3