These are the remaining issues found in /res.

ASTERISK-29761

Change-Id: I572e6019c422780dde5ce8448b6c85c77af6046d

Some modules have a different support level documented in their
MODULEINFO XML and Asterisk module definition. This change
brings the two in sync for the modules which were not matching.

ASTERISK-29336

Change-Id: If2f819103d4a271e2e0624ef4db365e897fa3d35

When dealing with a lot of video streams on WebRTC
the resulting SDPs can grow to be quite large. This
effectively doubles the maximum size to allow more
streams to exist.

The res_http_websocket module has also been changed
to use a buffer on the session for reading in packets
to ensure that the stack space usage is not excessive.

Change-Id: I31d4351d70c8e2c11564807a7528b984f3fbdd01

We read beyond the end of the buffer when copying the string out of the
buffer when we used ast_copy_string() because the original string was
not null terminated. Instead switch to ast_strndup() which does not
exhibit the same behavior.

ASTERISK-28975 #close

Change-Id: Ib4a75cffeb1eb8cf01136ef30306bd623e531a2a

Do not return error if the client received ping frame
while looking for a string and just wait for another frame.

ASTERISK-28958 #close

Change-Id: I4d06b4827bd71e56cbaafc011ffdcef9f0332922

ASTERISK-28949

Change-Id: Id465030f2b1997b83d408933fdbabe01827469ca

When TLS is in use, checking the readiness of the underlying FD is insufficient
for determining if there is data available to be read. So before polling the
FD, check if there is any buffered data in the TLS layer and use that first.

ASTERISK-28562 #close
Reported by: Robert Sutton

Change-Id: I95fcb3e2004700d5cf8e5ee04943f0115b15e10d

Control frames (PING / PONG / CLOSE) can be received in the middle of a
fragmented message. In order to ensure they do not interfere with the
reassembly buffer, we exit early and do not return the payload to the
caller.

ASTERISK-28257 #close

Change-Id: Ia5367144fe08ac6141bba3309517a48ec7f013bc

This ensures that Asterisk responds properly to frames received from a
client with opcode 8 (CLOSE) by echoing back the status code in its own
CLOSE frame.

Handling of the CLOSE opcode is moved up with the rest of the opcodes so
that unmasking gets applied. The payload is no longer returned to the
caller, but neither ARI nor the chan_sip nor pjsip made use of the
payload, which is a good thing since it was masked.

ASTERISK-28231 #close

Change-Id: Icb1b60205fc77ee970ddc91d1f545671781344cf

Replace usage of ao2_container_alloc with ao2_container_alloc_hash or
ao2_container_alloc_list.  Remove ao2_container_alloc macro.

Change-Id: I0907d78bc66efc775672df37c8faad00f2f6c088

The HTTP request processing in res_http_websocket allocates additional
space on the stack for various headers received during an Upgrade request.
An attacker could send a specially crafted request that causes this code
to overflow the stack, resulting in a crash.

* No longer allocate memory from the stack in a loop to parse the header
values.  NOTE: There is a slight API change when using the passed in
strings as is.  We now require the passed in strings to no longer have
leading or trailing whitespace.  This isn't a problem as the only callers
have already done this before passing the strings to the affected
function.

ASTERISK-28013 #close

Change-Id: Ia564825a8a95e085fd17e658cb777fe1afa8091a

* acl (named_acl.c)
* cdr
* cel
* ccss
* dnsmgr
* dsp
* enum
* extconfig (config.c)
* features
* http
* indications
* logger
* manager
* plc
* sounds
* udptl

These modules are now loaded at appropriate time by the module loader.
Unlike loadable modules these use AST_MODULE_LOAD_FAILURE on error so
the module loader will abort startup on failure of these modules.

Some of these modules are still initialized or shutdown from outside the
module loader.  logger.c is initialized very early and shutdown very
late, manager.c is initialized by the module loader but is shutdown by
the Asterisk core (too much uses it without holding references).

Change-Id: I371a9a45064f20026c492623ea8062d02a1ab97f

In ast_websocket_read() we were not adequately checking that the
payload_len was non-zero before passing it to ws_safe_read(). Calling
ws_safe_read with a len argument of 0 will result in a busy loop until
the underlying socket is closed.

ASTERISK-27658 #close

Change-Id: I9d59f83bc563f711df1a6197c57de473f6b0663a

Change-Id: Ic449ea832bc81a1671c0e910c5fbe8c683e3da89

This removes references that are no longer needed due to automatic
references created by module dependencies.

In addition this removes most calls to ast_module_check as they were
checking modules which are listed as dependencies.

Change-Id: I332a6e8383d4c72c8e89d988a184ab8320c4872e

clang 5.0 warned about this.

ASTERISK-27557

Change-Id: I7cceaa88e147cbdf81a3a7beec5c1c20210fa41e

There are many places in the code base where we ignore the return value
of fcntl() when getting/setting file descriptior flags. This patch
introduces a convenience function that allows setting or clearing file
descriptor flags and will also log an error on failure for later
analysis.

Change-Id: I8b81901e1b1bd537ca632567cdb408931c6eded7

Previously for PJSIP the local address of WebSocket connections
was set to the remote address. For logging purposes this is
not particularly useful.

The WebSocket API has been extended to allow the local
address to be queried and this is used in PJSIP to set the
local address to the correct value.

The PJSIP HEP support has also been tweaked so that reliable
transports always use the local address on the transport
and do not try to (wrongly) guess. As they are connection
based it is impossible for the source to be anything else.

ASTERISK-26758
ASTERISK-27363

Change-Id: Icd305fd038ad755e2682ab2786e381f6bf29e8ca

Once an Optional API module is loaded it should stay loaded.  Unloading
an optional API module runs the risk of a crash if something else is
using it.  This patch causes all optional API providers to tell the
module loader not to unload except at shutdown.

ASTERISK-27389

Change-Id: Ia07786fe655681aec49cc8d3d96e06483b11f5e6

In all non-pbx modules, AST_MODULE_LOAD_FAILURE has been changed
to AST_MODULE_LOAD_DECLINE.  This prevents asterisk from exiting
if a module can't be loaded.  If the user wishes to retain the
FAILURE behavior for a specific module, they can use the "require"
or "preload-require" keyword in modules.conf.

A new API was added to logger: ast_is_logger_initialized().  This
allows asterisk.c/check_init() to print to the error log once the
logger subsystem is ready instead of just to stdout.  If something
does fail before the logger is initialized, we now print to stderr
instead of stdout.

Change-Id: I5f4b50623d9b5a6cb7c5624a8c5c1274c13b2b25

When doing some WebRTC testing, I found that the websocket would
disconnect whenever I attempted to place a call into Asterisk. After
looking into it, I pinpointed the problem to be due to the iostreams
change being merged in.

Under certain circumstances, a call to ast_iostream_read() can return a
negative value. However, in this circumstance, the websocket code was
treating this negative return as if it were a partial read from the
websocket. The expected length would get adjusted by this negative
value, resulting in the expected length being too large.

This patch simply adds an if check to be sure that we are only updating
the expected length of a read when the return from a read is positive.

ASTERISK-26842 #close
Reported by Mark Michelson

Change-Id: Ib4423239828a013d27d7bc477d317d2f02db61ab

Expression 'rlen < 0' is always false.
Unsigned type value is never < 0.

Change-Id: Id9f393ff25b009a6c4a6e40b95f561a9369e4585

fopencookie/funclose is a non-standard API and should not be used
in portable software. Additionally, the way FILE's fd is used in
non-blocking mode is undefined behaviour and cannot be relied on.

This introduces internal abstraction for io streams, that allows
implementing the desired virtualization of read/write operations
with necessary timeout handling.

ASTERISK-24515 #close
ASTERISK-24517 #close

Change-Id: Id916aef418b665ced6a7489aef74908b6e376e85

Not surprisingly, using Respoke (and possibly other systems) it is
possible to blow past the 16k limit for a WebSocket packet size. This
patch bumps it up to 32k, which, at least for Respoke, is sufficient.
For now.

Because 32k is laughable on a LOW_MEMORY system (as is 16k, for that
matter), this patch adds a LOW_MEMORY directive that sets the buffer to
8k for systems who have asked for their reduced memory availability to
be considered.

Change-Id: Id235902537091b58608196844dc4b045e383cd2e

ASTERISK_REGISTER_FILE no longer has any purpose so this commit removes
all traces of it.

Previously exported symbols removed:
* __ast_register_file
* __ast_unregister_file
* ast_complete_source_filename

This also removes the mtx_prof static variable that was declared when
MTX_PROFILE was enabled.  This variable was only used in lock.c so it
is now initialized in that file only.

ASTERISK-26480 #close

Change-Id: I1074af07d71f9e159c48ef36631aa432c86f9966

Websockets are a core part of ARI support and as such this
module should also be core supported.

Change-Id: I8f9283c6a167152761b92984779bb39e3db51a9c

Updated ast_websocket_write to encode the entire frame in to one
write operation, to ensure that we don't end up with a situation
where the websocket header has been sent, while the body can not
be written.

Previous to August's patch in commit b9bd3c14, certain network
conditions could cause the header to be written, and then the
sub-sequent body to fail - which would cause the next successful
write to contain a new header, and a new body (resulting in
the peer receiving two headers - the second of which would be
read as part of the body for the first header).

This was patched to have both write operations individually fail
by closing the websocket.

In a case available to the submitter of this patch, the same
body which would consistently fail to write, would succeed
if written at the same time as the header.

This update merges the two operations in to one, adds debug messages
indicating the reason for a websocket connection being closed during
a write operation, and clarifies some variable names for code legibility.

Change-Id: I4db7a586af1c7a57184c31d3d55bf146f1a40598

Change-Id: I228df6adecd4cb450d03e09e9a38c86bb566e811

* Remove extraneous unlock on off-nominal path.
* Add missing HTTP error reply.

Change-Id: I1f402bfe448fba8696b507477cab5f060ccd9b2b

Due to the use of ast_websocket_close in session termination it is
possible for the underlying socket to already be closed when the
session is terminated. This occurs when the close frame is attempted
to be written out but fails.

Change-Id: I7572583529a42a7dc911ea77a974d8307d5c0c8b

The res_http_websocket module will currently attempt to close
the WebSocket connection if fatal cases occur, such as when
attempting to write out data and being unable to. When the
fatal cases occur the code attempts to write a WebSocket close
frame out to have the remote side close the connection. If
writing this fails then the connection is not terminated.

This change forcefully terminates the connection if the
WebSocket is to be closed but is unable to send the close frame.

ASTERISK-25312 #close

Change-Id: I10973086671cc192a76424060d9ec8e688602845

We don't have a compatability function to fill in a missing htobe64; but
we already have one for the identical htonll.

Change-Id: Ic0a95db1c5b0041e14e6b127432fb533b97e4cac

Commit 39cc28f6 attempted to fix a
test failure observed on 32 bit test agents by ensuring that a cast from
a 32 bit unsigned integer to a 64 bit unsigned integer was happening in
a predictable place. As it turns out, this did not cause test runs to
succeed.

This commit adds several redundant debug messages that print the payload
lengths of websocket frames. The idea here is that this commit will not
cause tests to succeed for the faulty test agent, but we might deduce
where the fault lies more easily this way by observing at what point the
expected value (537) changes to some ungangly huge number.

If you are wondering why something like this is being committed to the
branch, keep in mind that in commit
39cc28f6 I noted that the observed test
failures only happen when automated tests are run. Attempts to run the
tests by hand manually on the test agent result in the tests passing.

Change-Id: I14a65c19d8af40dadcdbd52348de3b0016e1ae8d

We have seen a rash of test failures on a 32-bit build agent. Commit
48698a5e solved an obvious problem where
we were not encoding a 64-bit value correctly over the wire. This
commit, however, did not solve the test failures.

In the failing tests, ARI is attempting to send a 537 byte text frame
over a websocket. When sending a frame this small, 16 bits are all that
is required in order to encode the payload length on the websocket
frame. However, ast_websocket_write() thinks that the payload length is
greater than 65535 and therefore writes out a 64 bit payload length.
Inspecting this payload length, the lower 32 bits are exactly what we
would expect it to be, 537 in hex. The upper 32 bits, are junk values
that are not expected to be there.

In the failure, we are passing the result of strlen() to a function that
expects a uint64_t parameter to be passed in. strlen() returns a size_t,
which on this 32-bit machine is 32 bits wide. Normally, passing a 32-bit
unsigned value to somewhere where a 64-bit unsigned value is expected
would cause no problems. In fact, in manual runs of failing tests, this
works just fine. However, ast_websocket_write() uses the Asterisk
optional API, which means that rather than a simple function call, there
are a series of macros that are used for its declaration and
implementation. These macros may be causing some sort of error to occur
when converting from a 32 bit quantity to a 64 bit quantity.

This commit changes the logic by making existing ast_websocket_write()
calls use ast_websocket_write_string() instead. Within
ast_websocket_write_string(), the 64-bit converted strlen is saved in a
local variable, and that variable is passed to ast_websocket_write()
instead.

Note that this commit message is full of speculation rather than
certainty. This is because the observed test failures, while always
present in automated test runs, never occur when tests are manually
attempted on the same test agent. The idea behind this commit is to fix
a theoretical issue by performing changes that should, at the least,
cause no harm. If it turns out that this change does not fix the failing
tests, then this commit should be reverted.

Change-Id: I4458dd87d785ca322b89c152b223a540a3d23e67

Prior to ASTERISK-24988, the WebSocket handshake was resolved before Stasis
applications were registered. This was done such that the WebSocket would be
ready when an application is registered. However, by creating the WebSocket
first, the client had the ability to make requests for the Stasis application
it thought had been created with the initial handshake request. The inevitable
conclusion of this scenario was the cart being put before the horse.

ASTERISK-24988 resolved half of the problem by ensuring that the applications
were created and registered with Stasis prior to completing the handshake
with the client. While this meant that Stasis was ready when the client
received the green-light from Asterisk, it also meant that the WebSocket was
not yet ready for Stasis to dispatch messages.

This patch introduces a message queuing mechanism for delaying messages from
Stasis applications while the WebSocket is being constructed. When the ARI
event processor receives the message from the WebSocket that it is being
created, the event processor instantiates an event session which contains a
message queue. It then tries to create and register the requested applications
with Stasis. Messages that are dispatched from Stasis between this point and
the point at which the event processor is notified the WebSocket is ready, are
stashed in the queue. Once the WebSocket has been built, the queue's messages
are dispatched in the order in which they were originally received and the
queue is concurrently cleared.

ASTERISK-25181 #close
Reported By: Matt Jordan

Change-Id: Iafef7b85a2e0bf78c114db4c87ffc3d16d671a17

A test agent was continuously failing all ARI tests when run against
Asterisk 13. As it turns out, the reason for this is that on those test
runs, for some reason we decided to use the super extended 64 bit
payload length for websocket text frames instead of the extended 16 bit
payload length. For 64-bit payloads, the expected byte order over the
network is

7, 6, 5, 4, 3, 2, 1, 0

However, we were sending the payload as

3, 2, 1, 0, 7, 6, 5, 4

This meant that we were saying to expect an absolutely MASSIVE payload
to arrive. Since we did not follow through on this expected payload
size, the client would sit patiently waiting for the rest of the payload
to arrive until the test would time out.

With this change, we use the htobe64() function instead of htonl() so
that a 64-bit byte-swap is performed instead of a 32 bit byte-swap.

Change-Id: Ibcd8552392845fbcdd017a8c8c1043b7fe35964a

This change makes it so that when accepting a WebSocket
connection the HTTP response is sent as one packet instead of
fragmented. Browsers don't like it when you send it fragmented.

ASTERISK-25103

Change-Id: I9b82c4ec2949b0bce692ad0bf6f7cea9709e7f69

This patch updates http_websocket and its corresponding implementation
with a pre-session established callback. This callback allows for
WebSocket server consumers to be notified when a WebSocket connection is
attempted, but before we accept it. Consumers can choose to reject the
connection, if their application specific logic allows for it.

As a result, this patch pulls out the previously private
websocket_protocol struct and makes it public, as
ast_websocket_protocol. In order to preserve backwards compatibility
with existing modules, the existing APIs were left as-is, and new APIs
were added for the creation of the ast_websocket_protocol as well as for
adding a sub-protocol to a WebSocket server.

In particular, the following new API calls were added:
* ast_websocket_add_protocol2 - add a protocol to the core WebSocket
  server
* ast_websocket_server_add_protocol2 - add a protocol to a specific
  WebSocket server
* ast_websocket_sub_protocol_alloc - allocate a sub-protocol object.
  Consumers can populate this with whatever callbacks they wish to
  support, then add it to the core server or a specified server.

ASTERISK-24988
Reported by: Joshua Colp

Change-Id: Ibe0bbb30c17eec6b578071bdbd197c911b620ab2

Change-Id: Icf88f9f861c6b2a16e5f626ff25795218a6f2723

Git does not support the ability to replace a token with a version
string during check-in. While it does have support for replacing a
token on clone, this is somewhat sub-optimal: the token is replaced
with the object hash, which is not particularly easy for human
consumption. What's more, in practice, the source file version was often
not terribly useful. Generally, when triaging bugs, the overall version
of Asterisk is far more useful than an individual SVN version of a file. As a
result, this patch removes Asterisk's support for showing source file
versions.

Specifically, it does the following:

* Rename ASTERISK_FILE_VERSION macro to ASTERISK_REGISTER_FILE, and
  remove passing the version in with the macro. Other facilities
  than 'core show file version' make use of the file names, such as
  setting a debug level only on a specific file. As such, the act of
  registering source files with the Asterisk core still has use. The
  macro rename now reflects the new macro purpose.

* main/asterisk:
  - Refactor the file_version structure to reflect that it no longer
    tracks a version field.
  - Remove the "core show file version" CLI command. Without the file
    version, it is no longer useful.
  - Remove the ast_file_version_find function. The file version is no
    longer tracked.
  - Rename ast_register_file_version/ast_unregister_file_version to
    ast_register_file/ast_unregister_file, respectively.

* main/manager: Remove value from the Version key of the ModuleCheck
  Action. The actual key itself has not been removed, as doing so would
  absolutely constitute a backwards incompatible change. However, since
  the file version is no longer tracked, there is no need to attempt to
  include it in the Version key.

* UPGRADE: Add notes for:
  - Modification to the ModuleCheck AMI Action
  - Removal of the "core show file version" CLI command

Change-Id: I6cf0ff280e1668bf4957dc21f32a5ff43444a40e