Some optimizations in regards to downloading Favicon's.
I also encounterd some issues with accessing some sites where the
connection got dropped or closed early. This seems a reqwest/hyper
thingy, https://github.com/hyperium/hyper/issues/2136. This is now also
fixed.
General:
- Decreased struct size
- Decreased memory allocations
- Optimized tokenizer a bit more to only emit tags when all attributes are there and are valid.
reqwest/hyper connection issue:
The following changes helped solve the connection issues to some sites.
The endresult is that some icons are now able to be downloaded always instead of sometimes.
- Enabled some extra reqwest features, `deflate` and `native-tls-alpn`
(Which do not bring in any extra crates since other crates already enabled them, but they were not active for Vaultwarden it self)
- Configured reqwest to have a max amount of idle pool connections per host
- Configured reqwest to timeout the idle connections in 10 seconds
During setting the latest commit hash for Rocket and updating all the
other crates, there were some messages regarding the usage of `String`
for the Rocket endpoint function calls. I acted upon this message and
changed all `String` types to `&str` and modified the code where needed.
This ended up in less alloc calls, and probably also a bit less memory usage.
- Updated all the crates and commit hashes
- Modified all `String` to `&str` where applicable
When an icon will not be downloaded due to matching a configured
blacklist, ensure that the log message indicates the type of blacklist
that was matched.
- Several cleanups and code optimizations for Emergency Access
- Fixed a race-condition regarding jobs for Emergency Access
- Some other small changes like `allow(clippy::)` removals
Fixes #2925
When a icon blacklist regex was configured to not check for a domain, it
still did a DNS lookup first. This could cause a DNS leakage for these
regex blocked domains.
This PR resolves this issue by first checking the regex, and afterwards
the other checks.
Fixes #2909
When using anything else but the `internal` icon service it would
trigger an CSP block because the redirects were not allowed.
This PR fixes #2623 by dynamically adding the needed CSP strings.
This should also work with custom services.
For Google i needed to add an extra check because that does a redirect
it self to there gstatic.com domain.
A bit inspired by @paolobarbolini from this commit at lettre https://github.com/lettre/lettre/pull/784 .
I added a few more clippy lints here, and fixed the resulted issues.
Overall i think this could help in preventing future issues, and maybe
even peformance problems. It also makes some code a bit more clear.
We could always add more if we want to, i left a few out which i think
arn't that huge of an issue. Some like the `unused_async` are nice,
which resulted in a few `async` removals.
Some others are maybe a bit more estatic, like `string_to_string`, but i
think it looks better to use `clone` in those cases instead of `to_string` while they already are a string.
- Updated dependencies (html5gum for favicon downloading)
* Also openssl, time, jsonwebtoken and r2d2
- Small optimizations on downloading favicons.
It now only emits tokens/tags which needs to be parsed, all others are
being skipped. This prevents unneeded items within the for-loop being
parsed.
Favicon:
- Replaced HTML tokenizer, much faster now.
- Caching the domain blacklist function.
- Almost all functions are async now.
- Fixed bug on minimizing data to parse
- Changed maximum icon download size to 5MB to match Bitwarden
- Added `apple-touch-icon.png` as a second fallback besides `favicon.ico`
SMTP:
- Deprecated SMTP_SSL and SMTP_EXPLICIT_TLS, replaced with SMTP_SECURITY
Misc:
- Fixed issue when `resolv.conf` contains errors and trust-dns panics (Fixes #2283)
- Updated Javscript and CSS files for admin interface
- Fixed an issue with the /admin interface which did not cleared the login cookie correctly
- Prevent websocket notifications during org import, this caused a lot of traffic, and slowed down the import.
This is also the same as Bitwarden which does not trigger this refresh via websockets.
Rust:
- Updated to use v1.59
- Use the new `strip` option and enabled to strip `debuginfo`
- Enabled `lto` with `thin`
- Removed the strip RUN from the alpine armv7, this is now done automatically
This is a rather large PR which updates the async branch to have all the
database methods as an async fn.
Some iter/map logic needed to be changed to a stream::iter().then(), but
besides that most changes were just adding async/await where needed.
The default code is 307 (temporary) to make it easier to test different icon
services, but once a service has been decided on, users should ideally switch
to using permanent redirects for cacheability.
Browsers are rather smart, but also dumb. This uses the `Expires` header
alongside `cache-control` to better prompt the browser to actually
cache.
Unfortunately, firefox still tries to "race" its own cache, in an
attempt to respond to requests faster, so still ends up making a bunch
of requests which could have been cached. Doesn't appear there's any way
around this.
If an external icon service is configured, icon requests return an HTTP
redirect to the corresponding icon at the external service.
An external service may be useful for various reasons, such as if:
* The Vaultwarden instance has no external network connectivity.
* The Vaultwarden instance has trouble handling large bursts of icon requests.
* There are concerns that an attacker may probe the instance to try to detect
whether icons for certain sites have been cached, which would suggest that
the instance contains entries for those sites.
* The external icon service does a better job of providing icons than the
built-in fetcher.
- Unicode websites could break (www.post.japanpost.jp for example).
regex would fail because it was missing the unicode-perl feature.
- Be less verbose in logging with icon downloads
- Removed duplicate info/error messages
- Added err_silent! macro to help with the less verbose error/info messages.
- Updated some crates
- Updated icon fetching code:
+ Use a cookie jar and set Max-Age to 2 minutes for all cookies
+ Locate the base href tag to fix some locations
+ Changed User-Agent (Helps on some sites to get HTML instead of JS)
+ Reduced HTML code limit from 512KB to 384KB
+ Allow some large icons higer-up in the sort
+ Allow GIF images
+ Ignore cookie_store and hyper::client debug messages
- Updated branding for admin and emails
- Updated crates and some deprications
- Removed newline-converter because this is built-in into lettre
- Updated email templates to use a shared header and footer template
- Also trigger SMTP SSL When TLS is selected without SSL
Resolves #1641
- Added image type checking, and prevent downloading non images.
We didn't checked this before, which could in turn could allow someone
to download an arbitrary file.
- This also prevents SVG images from being used, while they work on the
web-vault and desktop client, they didn't on the mobile versions.
- Because of this image type checking we can return a valid file type
instead of only 'x-icon' (which is still used as a fallback).
- Prevent rel values with `icon-mask`, these are not valid favicons.
- We need to add some feature to enable smtp debugging again. See: https://github.com/lettre/lettre/pull/584
- Upstream added the fallback icon again, probably because of caching ;). See: https://github.com/bitwarden/server/pull/1149
- Enabled gzip and brotli compression support with reqwest. Some sites seem to force this, or assume that because of the User-Agent string it is supported. This caused some failed icons.
Fixes #1540
- Changed the user-agent, which caused at least one site to stall the
connection (Same happens on icons.bitwarden.com)
- Added default_header creation to the lazy static CLIENT
- Added referer passing, which is checked by some sites
- Some small other changes
- Added more checks to prevent panics (Removed unwrap)
- Try do download from base domain or add www when the provided domain
fails
- Added some more domain validation checks to prevent errors
- Added the ICON_BLACKLIST_REGEX to a Lazy Static HashMap which
speeds-up the checks!
- Validate the Regex before starting/config change.
- Some cleanups
- Disabled some noisy debugging from 2 crates.
