From ac032544bf2c09ba8935128b8cbd6a669a1b91df Mon Sep 17 00:00:00 2001
From: Andy Green <andy@warmcat.com>
Date: Fri, 16 Nov 2018 09:10:41 +0800
Subject: [PATCH] README.md: update in preparation for v3.1
---
README.md | 102 ++++++++++++++++++----
READMEs/README.content-security-policy.md | 33 +++++--
READMEs/README.test-apps.md | 10 ++-
changelog | 77 +++++++++++++++-
lib/misc/fts/README.md | 3 +-
5 files changed, 195 insertions(+), 30 deletions(-)
diff --git a/README.md b/README.md
index 1e7ab63d..6f919b79 100644
--- a/README.md
+++ b/README.md
@@ -1,10 +1,81 @@
[](https://travis-ci.org/warmcat/libwebsockets) [](https://ci.appveyor.com/project/lws-team/libwebsockets) [](https://scan.coverity.com/projects/3576) [](https://bestpractices.coreinfrastructure.org/projects/2266) [](https://www.codacy.com/app/lws-team/libwebsockets?utm_source=github.com&utm_medium=referral&utm_content=warmcat/libwebsockets&utm_campaign=Badge_Grade)
+# Libwebsockets
+
+Libwebsockets is a simple-to-use, pure C library providing client and server
+for **http/1**, **http/2**, **websockets** and other protocols in a security-minded,
+lightweight, configurable, scalable and flexible way. It's easy to build and
+cross-build via cmake and is suitable for tasks from embedded RTOS through mass
+cloud serving.
+
+[50 minimal examples](https://libwebsockets.org/git/libwebsockets/tree/minimal-examples) for various scenarios, CC0-licensed (public domain) for cut-and-paste, allow you to get started quickly.
+
News
----
+## New features in v3.1
+
+ - **lws threadpool** - lightweight pool of pthreads integrated to lws wsi, with all
+ synchronization to event loop handled internally, queue for excess tasks
+ [threadpool docs](https://libwebsockets.org/git/libwebsockets/tree/lib/misc/threadpool)
+ [threadpool minimal example](https://libwebsockets.org/git/libwebsockets/tree/minimal-examples/ws-server/minimal-ws-server-threadpool)
+ Cmake config: `-DLWS_WITH_THREADPOOL=1`
+
+ - **libdbus support** integrated on lws event loop
+ [lws dbus docs](https://libwebsockets.org/git/libwebsockets/tree/lib/roles/dbus)
+ [lws dbus client minimal examples](https://libwebsockets.org/git/libwebsockets/tree/minimal-examples/dbus-client)
+ [lws dbus server minimal examples](https://libwebsockets.org/git/libwebsockets/tree/minimal-examples/dbus-server)
+ Cmake config: `-DLWS_ROLE_DBUS=1`
+
+ - **lws allocated chunks (lwsac)** - helpers for optimized mass allocation of small
+ objects inside a few larger malloc chunks... if you need to allocate a lot of
+ inter-related structs for a limited time, this removes per-struct allocation
+ library overhead completely and removes the need for any destruction handling
+ [lwsac docs](https://libwebsockets.org/git/libwebsockets/tree/lib/misc/lwsac)
+ [lwsac minimal example](https://libwebsockets.org/git/libwebsockets/tree/minimal-examples/api-tests/api-test-lwsac)
+ Cmake Config: `-DLWS_WITH_LWSAC=1`
+
+ - **lws tokenizer** - helper api for robustly tokenizing your own strings without
+ allocating or adding complexity. Configurable by flags for common delimiter
+ sets and comma-separated-lists in the tokenizer. Detects and reports syntax
+ errors.
+ [lws_tokenize docs](https://libwebsockets.org/git/libwebsockets/tree/include/libwebsockets/lws-tokenize.h)
+ [lws_tokenize minimal example / api test](https://libwebsockets.org/git/libwebsockets/tree/minimal-examples/api-tests/api-test-lws_tokenize)
+
+ - **lws full-text search** - optimized trie generation, serialization,
+ autocomplete suggestion generation and instant global search support extensible
+ to huge corpuses of UTF-8 text while remaining super lightweight on resources.
+ [full-text search docs](https://libwebsockets.org/git/libwebsockets/tree/lib/misc/fts)
+ [full-text search minimal example / api test](https://libwebsockets.org/git/libwebsockets/tree/minimal-examples/api-tests/api-test-fts)
+ [demo](https://libwebsockets.org/ftsdemo/)
+ [demo sources](https://libwebsockets.org/git/libwebsockets/tree/plugins/protocol_fulltext_demo.c)
+ Cmake config: `-DLWS_WITH_FTS=1 -DLWS_WITH_LWSAC=1`
+
+ - **gzip + brotli http server-side compression** - h1 and h2 detection of client support
+ for server compression, and auto-application to files with mimetypes "text/*",
+ "application/javascript" and "image/svg.xml".
+ Cmake config: `-DLWS_WITH_HTTP_STREAM_COMPRESSION=1` for gzip, optionally also give
+ `-DLWS_WITH_HTTP_BROTLI=1` for preferred `br` brotli compression
+
+ - **managed disk cache** - API for managing a directory containing cached files
+ with hashed names, and automatic deletion of LRU files once the cache is
+ above a given limit.
+ [lws diskcache docs](https://libwebsockets.org/git/libwebsockets/tree/include/libwebsockets/lws-diskcache.h)
+ Cmake config: `-DLWS_WITH_DISKCACHE=1`
+
+ - **http reverse proxy** - lws mounts support proxying h1 or h2 requests to
+ a local or remote IP, or unix domain socket over h1. This allows microservice
+ type architectures where parts of the common URL space are actually handled
+ by external processes which may be remote or on the same machine.
+ [lws gitohashi serving](https://libwebsockets.org/git/) is handled this way.
+ [unix domain sockets reverse proxy docs](https://libwebsockets.org/git/libwebsockets/tree/READMEs/README.unix-domain-reverse-proxy.md)
+ CMake config: `-DLWS_WITH_HTTP_PROXY=1`
+
+ - **release policy docs** - unsure what branch, version or tag to use, or how
+ to follow master cleanly? [Read the release policy docs](https://libwebsockets.org/git/libwebsockets/tree/READMEs/README.release-policy.md)
+
## v3.0.1 released
See the git log for the list of fixes.
@@ -15,14 +86,14 @@ See the changelog for info https://libwebsockets.org/git/libwebsockets/tree/chan
## Major CI improvements for QA
-The Travis build of lws done on every commit now runs
+The Travis build of lws done on every commit now runs:
Tests|Count|Explanation
---|---|---
-Build / Linux / gcc|14|-Wall -Werror cmake config variants
-Build / Mac / Clang|14|-Wall -Werror cmake config variants
+Build / Linux / gcc|16|-Wall -Werror cmake config variants
+Build / Mac / Clang|16|-Wall -Werror cmake config variants
Build / Windows / MSVC|7|default
-Selftests|openssl:33, mbedtls:33|minimal examples built and run against each other and remote server
+Selftests|openssl:43, mbedtls:43|minimal examples built and run against each other and remote server
attack.sh|225|Correctness, robustness and security tests for http parser
Autobahn Server|480|Testing lws ws client, including permessage-deflate
Autobahn Client|480|Testing lws ws server, including permaessage-deflate
@@ -30,9 +101,13 @@ h2spec|openssl:146, mbedtls:146|Http/2 server compliance suite (in strict mode)
h2load|openssl:6, mbedtls:6|Http/2 server load tool (checks 10K / 100K in h1 and h2, at 1, 10, 100 concurrency)
h2load SMP|6|Http/2 and http/1.1 server load checks on SMP server build
-The over 1,500 tests run on every commit take most of an hour to complete.
+The over 1,500 tests run on every commit take 1hr 15 of compute time to complete.
If any problems are found, it breaks the travis build, generating an email.
+Codacy also checks every patch and the information used to keep lws at zero issues.
+
+Current master is checked by Coverity at least daily and kept at zero issues.
+
Current master passes all the tests and these new CI arrangements will help
keep it that way.
@@ -40,7 +115,7 @@ keep it that way.
-There's a new standard on the RFC track that enables multiplexing ws connections
+There's a new [RFC](https://tools.ietf.org/html/rfc8441) that enables multiplexing ws connections
over an http/2 link. Compared to making individual tcp and tls connections for
each ws link back to the same server, this makes your site start up radically
faster, and since all the connections are in one tls tunnel, with considerable memory
@@ -53,8 +128,9 @@ or to take advantage of ws-over-h2. When built with http/2 support, it automati
falls back to http/1 and traditional ws upgrade if that's all the client can handle.
Currently only Chrome Canary v67 supports this ws-over-h2 encapsulation (chrome
-must be started with `--enable-websocket-over-http2` switch to enable it currently)
-but the other browsers will catch up soon.
+must be started with `--enable-websocket-over-http2` switch to enable it currently),
+and patches exist for Firefox. Authors of both browser implementations tested
+against the lws server implementation.
## New "minimal examples"
@@ -73,16 +149,6 @@ are CC0 licensed (public domain) to facilitate that.
[lws on Appveyor](https://ci.appveyor.com/project/lws-team/libwebsockets),
click on a build, the ARTIFACTS, and unzip the zip file at `C:\Program Files (x86)/libwebsockets`.
-## ESP32 is supported
-
-ESP32 is now supported in lws! Download the
-
- - factory https://warmcat.com/git/lws-esp32-factory/ and
- - test server app https://warmcat.com/git/lws-esp32-test-server-demos
-
-The ESP32 stuff has my dynamic mbedtls buffer allocation patches applied,
-which reduce allocation for small payload TLS links by around 26KiB per connection.
-
## Support
This is the libwebsockets C library for lightweight websocket clients and
diff --git a/READMEs/README.content-security-policy.md b/READMEs/README.content-security-policy.md
index 07e04daa..4514a67d 100644
--- a/READMEs/README.content-security-policy.md
+++ b/READMEs/README.content-security-policy.md
@@ -47,10 +47,25 @@ minimum needed for the pages to operate.
necessary so everything in the HTML can stay the same... it means adapt the
pages to want the minimum and then enable the minimum.
+The main point is segregation of styles and script away from the content, in
+files referenced in the document `<head>` section, along these lines:
+
+```
+<head>
+ <meta charset=utf-8 http-equiv="Content-Language" content="en"/>
+ <link rel="stylesheet" type="text/css" href="test.css"/>
+ <script type='text/javascript' src="/lws-common.js"></script>
+ <script type='text/javascript' src='test.js'></script>
+ <title>Minimal Websocket test app</title>
+</head>
+```
+
#### Inline styles must die
-All styling must go in one or more `.css` file served by the same server
-(preferably... you can whitelist other sources in the CSP if you have to).
+All styling must go in one or more `.css` file(s) best served by the same
+server... while you can whitelist other sources in the CSP if you have to,
+unless you control that server as well, you are allowing whoever gains
+access to that server access to your users.
Inline styles are no longer allowed (eg, "style='font-size:120%'" in the
HTML)... they must be replaced by reference to one or more CSS class, which
@@ -60,14 +75,14 @@ practice anyway, and your pages will be cleaner and more maintainable.
#### Inline scripts must die
Inline scripts need to be placed in a `.js` file and loaded in the page head
-section, from the server that provided the page.
+section, again it should only be from the server that provided the page.
Then, any kind of inline script, yours or injected or whatever, will be
completely rejected by the browser.
#### onXXX must be replaced by eventListener
-Inline `onclick` or whatever are kinds of inline scripting and are banned.
+Inline `onclick()` etc are kinds of inline scripting and are banned.
Modern browsers have offered a different system called ["EventListener" for
a while](https://developer.mozilla.org/en-US/docs/Web/API/EventListener) which allows binding of events to DOM elements in JS.
@@ -118,9 +133,15 @@ it, hosting them on "self", ie, the same server that provided the HTML,
will remove that problem.
Bringing in scripts from external sources is actually quite scary from the
-security perspective. If someone hacks the jquery site to serve a hostile
-payload in addition to the usual scripts, half the Internet will instantly
+security perspective. If someone hacks the `ajax.googleapis.com` site to serve
+a hostile, modified jquery, half the Internet will instantly
become malicious. However if you serve it yourself, unless your server
was specifically targeted you know it will continue to serve what you
expect.
+Since these scripts are usually sent with cache control headers for local
+caching duration of 1 year, the cost of serving them yourself under the same
+conditions is small but your susceptibility to attack is reduced to only taking
+care of your own server. And there is a privacy benefit that google is not
+informed of your users' IPs and activities on your site.
+
diff --git a/READMEs/README.test-apps.md b/READMEs/README.test-apps.md
index 4482ec8b..7565b43f 100644
--- a/READMEs/README.test-apps.md
+++ b/READMEs/README.test-apps.md
@@ -391,8 +391,8 @@ version 13.
Since libwebsockets runs using `poll()` and a single threaded approach, any
unexpected latency coming from system calls would be bad news. There's now
-a latency tracking scheme that can be built in with `--with-latency` at
-configure-time, logging the time taken for system calls to complete and if
+a latency tracking scheme that can be built in with `-DLWS_WITH_LATENCY=1` at
+cmake, logging the time taken for system calls to complete and if
the whole action did complete that time or was deferred.
You can see the detailed data by enabling logging level 512 (eg, `-d 519` on
@@ -417,7 +417,11 @@ server modes
1) pip install autobahntestsuite
-2) From your build dir: cmake .. -DLWS_WITH_MINIMAL_EXAMPLES=1 && make
+2) From your build dir:
+
+```
+ $ cmake .. -DLWS_WITHOUT_EXTENSIONS=0 -DLWS_WITH_MINIMAL_EXAMPLES=1 && make
+```
3) ../scripts/autobahn-test.sh
diff --git a/changelog b/changelog
index 0931c892..e3ce5f2c 100644
--- a/changelog
+++ b/changelog
@@ -1,11 +1,86 @@
Changelog
---------
+v3.1.0
+======
+
- CHANGE: REMOVED: lws_client_connect() and lws_client_connect_extended()
compatibility apis for lws_client_connect_via_info() have been marked as
deprecated for several versions and are now removed. Use
- lws_client_connect_via_info() instead.
+ lws_client_connect_via_info() directly instead.
+
+ - CHANGE: CMAKE:
+ - LWS_WITH_HTTP2: now defaults ON
+ - NEW: CMAKE
+ - LWS_FOR_GITOHASHI: sets various cmake options suitable for gitohashi
+ - LWS_WITH_ASAN: for Linux, enable build with ASAN
+
+ Don't forget LWS_WITH_DISTRO_RECOMMENDED, which enables a wide range of lws
+ options suitable for a distro build of the library.
+
+ - NEW: lws threadpool - lightweight pool of pthreads integrated to lws wsi, with
+ all synchronization to event loop handled internally, queue for excess tasks
+ [threadpool docs](https://libwebsockets.org/git/libwebsockets/tree/lib/misc/threadpool)
+ [threadpool minimal example](https://libwebsockets.org/git/libwebsockets/tree/minimal-examples/ws-server/minimal-ws-server-threadpool)
+ Cmake config: `-DLWS_WITH_THREADPOOL=1`
+
+ - NEW: libdbus support integrated on lws event loop
+ [lws dbus docs](https://libwebsockets.org/git/libwebsockets/tree/lib/roles/dbus)
+ [lws dbus client minimal examples](https://libwebsockets.org/git/libwebsockets/tree/minimal-examples/dbus-client)
+ [lws dbus server minimal examples](https://libwebsockets.org/git/libwebsockets/tree/minimal-examples/dbus-server)
+ Cmake config: `-DLWS_ROLE_DBUS=1`
+
+ - NEW: lws allocated chunks (lwsac) - helpers for optimized mass allocation of small
+ objects inside a few larger malloc chunks... if you need to allocate a lot of
+ inter-related structs for a limited time, this removes per-struct allocation
+ library overhead completely and removes the need for any destruction handling
+ [lwsac docs](https://libwebsockets.org/git/libwebsockets/tree/lib/misc/lwsac)
+ [lwsac minimal example](https://libwebsockets.org/git/libwebsockets/tree/minimal-examples/api-tests/api-test-lwsac)
+ Cmake Config: `-DLWS_WITH_LWSAC=1`
+
+ - NEW: lws tokenizer - helper api for robustly tokenizing your own strings without
+ allocating or adding complexity. Configurable by flags for common delimiter
+ sets and comma-separated-lists in the tokenizer. Detects and reports syntax
+ errors.
+ [lws_tokenize docs](https://libwebsockets.org/git/libwebsockets/tree/include/libwebsockets/lws-tokenize.h)
+ [lws_tokenize minimal example / api test](https://libwebsockets.org/git/libwebsockets/tree/minimal-examples/api-tests/api-test-lws_tokenize)
+
+ - NEW: lws full-text search - optimized trie generation, serialization,
+ autocomplete suggestion generation and instant global search support extensible
+ to huge corpuses of UTF-8 text while remaining super lightweight on resources.
+ [full-text search docs](https://libwebsockets.org/git/libwebsockets/tree/lib/misc/fts)
+ [full-text search minimal example / api test](https://libwebsockets.org/git/libwebsockets/tree/minimal-examples/api-tests/api-test-fts)
+ [demo](https://libwebsockets.org/ftsdemo/)
+ [demo sources](https://libwebsockets.org/git/libwebsockets/tree/plugins/protocol_fulltext_demo.c)
+ Cmake config: `-DLWS_WITH_FTS=1 -DLWS_WITH_LWSAC=1`
+
+ - NEW: gzip + brotli http server-side compression - h1 and h2 automatic advertising
+ of server compression and application to files with mimetypes "text/*",
+ "application/javascript" and "image/svg.xml".
+ Cmake config: `-DLWS_WITH_HTTP_STREAM_COMPRESSION=1`, `-DLWS_WITH_HTTP_BROTLI=1`
+
+ - NEW: managed disk cache - API for managing a directory containing cached files
+ with hashed names, and automatic deletion of LRU files once the cache is
+ above a given limit.
+ [lws diskcache docs](https://libwebsockets.org/git/libwebsockets/tree/include/libwebsockets/lws-diskcache.h)
+ Cmake config: `-DLWS_WITH_DISKCACHE=1`
+
+ - NEW: http reverse proxy - lws mounts support proxying h1 or h2 requests to
+ a local or remote IP, or unix domain socket over h1. This allows microservice
+ type architectures where parts of the common URL space are actually handled
+ by external processes which may be remote or on the same machine.
+ [lws gitohashi serving](https://libwebsockets.org/git/) is handled this way.
+ CMake config: `-DLWS_WITH_HTTP_PROXY=1`
+
+ - NEW: lws_buflist - internally several types of ad-hoc malloc'd buffer have
+ been replaced by a new, exported api `struct lws_buflist`. This allows
+ multiple buffers to be chained and drawn down in strict FIFO order.
+
+ - NEW: In the case of h1 upgrade, the connection header is checked to contain
+ "upgrade". The vhost flag LWS_SERVER_OPTION_VHOST_UPG_STRICT_HOST_CHECK
+ also causes the Host: header to be confirmed to match the vhost name and
+ listen port.
v3.0.0
======
diff --git a/lib/misc/fts/README.md b/lib/misc/fts/README.md
index 7b151b5e..ef53abbf 100644
--- a/lib/misc/fts/README.md
+++ b/lib/misc/fts/README.md
@@ -19,8 +19,7 @@ around structures in the on-disk index file.
Function|Related Link
---|---
-Public API|[include/libwebsockets/lws-fts.h](
-https://libwebsockets.org/git/libwebsockets/tree/include/libwebsockets/lws-fts.h)
+Public API|[include/libwebsockets/lws-fts.h](https://libwebsockets.org/git/libwebsockets/tree/include/libwebsockets/lws-fts.h)
CI test app|[minimal-examples/api-tests/api-test-fts](https://libwebsockets.org/git/libwebsockets/tree/minimal-examples/api-tests/api-test-fts)
Demo minimal example|[minimal-examples/http-server/minimal-http-server-fulltext-search](https://libwebsockets.org/git/libwebsockets/tree/minimal-examples/http-server/minimal-http-server-fulltext-search)
Live Demo|[https://libwebsockets.org/ftsdemo/](https://libwebsockets.org/ftsdemo/)
--
GitLab