Also backported tests from our stand-alone Json lib to tolerate subtle
differences between simdjson and our backend. The differences are
really minor:
1. simdjson tolerates invalid unicode codepoint escapes (\udd6a) whereas our
lib is more restrictive.
2. simdjson has a maximum nesting level of 1024 and our lib has 512.
In either case these subtleties are not a problem in practice -- just that the
tests needed to be updated to tolerate these differences.
This test is a stand-alone program that tests performance for writing
and reading ot a dummy rocksdb database. It also is a good way to see
just how much memory rocksdb uses during normal operation.
I modified this test a little bit to use multiple threads when reading
and also I cleaned up the code a little bit.
This is a small CLI program that attempts to fragment the heap by
allocating a bunch of random data from multiple threads. It then
stops and asks the user to hit enter.
It prints memory usage for physical and virtual memory used by the
process, and at the end it compares the amount of memory still mapped
(even after deallocating all buffers).
It can be used as a crude way to estimate the efficiency of an allocator
and how good it is at releasing memory to the OS.
- Added a "fast path" to the JSON parser for strings (""). The optimistic case is that the string data is just pure ascii, with no escape chars. In that case, we simply take a shallow copy of the buffer we are currently parsing. This avoids using the (slow) `JSONUTF8StringFilter` to process unicode and unicode escapes (which does `push_back()` per character). The fast path offers a 10% or more speedup on JSON parsing (as well as fewer mallocs -> less heap fragmentation over time).
- JSON parser takes shallow copies of string data whenever it can do so (conceptually similar to `std::string_view`), to avoid on redundant copies and mallocs.
- Overall JSON parser is ~15% faster now.
- Fixed the correctness bug where unicode code point `\u0000` would end up being interpreted as end-of-string. We can now pass the original UniValue lib's `round1.json` test. This bug was due to the way `QString::fromUtf8` worked for `QByteArray`s. It turns out passing QString a raw C pointer with a size param is better. See: [This line of code](ba3b53cb50/src/corelib/text/qstring.h (L701)) versus [this line of code](ba3b53cb50/src/corelib/text/qstring.h (L709)).
- Added more corner-case JSON tests
- We also set the C++ locale and the Qt default `QLocale` to `"C"` as well on startup.. just in case.
- JSON Number parse was needlessly pushing 1 character at a time to the
output QByteArray. We instead just accept characters until the end
of token, then push the entire token as 1 call. This avoids extra
mallocs and extra function calls.
- Fixed a bug where the parser would accept bare "-" by itself as if it
were a number, but then fail later.
- Optimized some expressions slightly.
- Fixed some read-past-end-of-buffer bugs (in practice would not have
caused problems since they would technically be reading the NUL byte).
- Added more tests for corner cases (such as the '-' situation described
above).
- Made the Json_Parser.cpp "Container" struct store the parsed
QByteArray directly rather than it being wastefully wrapped by a
QVariant.
- Added a note-to-self about why we didn't use a union class (it bought us
no performance to do so and it just added boilerplate code). Maybe in
the future when we have std::variant on all compilers we target that
may be a decent alternative.
These tests were taken from the BCHN UniValue lib's tests. They are
pretty good. Can be run if compiling with -DENABLE_TESTS and then
specifying --test json on the CLI.
I have been using this/will use this to evaluate options that reduce
memory consumption.
Also added a blurb/description to the quick-config regarding
db_max_open_files.
All the rocksdb headers have been moved to the staticlibs folder
to prevent include path conflicts. They have also been removed
from the project file as they are not necessary for building and
when using a system rocksdb the headers are different ones.
Storage.cpp has been changed to include rocksdb from the system
include path first.
In the project there are new feature flags, the first of them
being "staticlibs" and set by default. These can be overridden
from the qmake command line like this:
qmake -makefiles features= Fulcrum.pro
The project now only adds the staticlibs to the include and
library path when the feature is set.
