Idea: sending binary/array data between scripts
tracked
Frio Belmonte
A use case that I have in mind: building an array of values in one script that another script needs to process. An encoding step is necessary since linkset message/linkset data/say strings can't contain nulls (I sure wish they COULD but that's probably too much to ask). Building it into a binary buffer or string is not flexible (need to know exact structure and length ahead of time) and doesn't support all basic data types directly. Building a string awkward and inefficient. Table.concat can't handle all types either, and a big CSV string is very much not compact and can't be string.split to original data types at the receiving end. lljson+llbase64 can of course handle a typed array already, but that's even less compact.
So my idea is: an encoder and decoder in llbase64 that take/produce an array-like, contiguous and flat table that supports all basic data types.
llbase64.encodearray(t:{}, start:number?, end:number?): string
- handle each table "t" element from "start" (defaults to 1) to "end" (defaults to #t) similar to string.pack producing the most condensed binary representation tagged with a type byte (e.g. "0" for nil, "t" for boolean true, "f" for boolean false, "n<8 bytes>" for a number, "s<length><data>" for strings, "v<8 bytes>" for vectors, etc... I don't remember how lljson did the type tagging, but maybe use the same tags), then base64 it. Errors on failure (table is not flat, contains userdata/threads/whatever else).
llbase64.decodearray(s:string, base_index:number?, limit:number?): {}
- decode base64 string "s" to a binary string and produce an array-like table starting at index "base_index" (defaults to 1), parsing at most "limit" (defaults to no limit) entries. Errors on failure.
Edit: since this was moved to Scripting Features from the Lua section, I'll just point out this is definitely a Lua request foremost: LSL is extremely limited with dealing with any kinda binary data and lacks the llbase64 library, even though the same logic could be applied to lists instead of array-like tables.
Log In
SuzannaLinn Resident
In SLua, we can use
lljson.slencode()
/ lljson.sldecode()
.The encoded string is safe to use with linkset data and linked messages. Buffers are base64 encoded and unsafe characters in strings are escaped. A null in a string becomes "/u0000".
This is the internal tagging: https://suzanna-linn.github.io/slua/moving-lljson#codes
Frio Belmonte
SuzannaLinn Resident Thanks, though my idea was more about something that's more predictable and hopefully faster to process. Fixed binary pack+base64 lengths per entry won't win over CSV and JSON in compactness all the time (shorter numbers get shorter text representations, of course), but you'd at least have a better idea of how much data you get beforehand and ideally it'd be fast to encode/decode.
On a quick test (table of 256 random 32-bit values, or a 1024 byte buffer for the same values), JSON encoder is ~4x slower than just dumping as CSV and ~100x slower than base64 encoding a buffer; the JSON decoder is ~1.4x slower than than string.splitting CSV back into a table, and ~17x slower than base64 decoding a string back into a buffer.
Memory used by json and CSV for just a numeric array is almost identical at ~2700 bytes (random numbers after all), the proposed binary pack+base64 format would use 3072 always as it encodes entire 64-bit numbers, not just 32-bit values, and an encoded buffer is the clear winner here too with 1368 bytes. Buffers being fixed size and not supporting all the data types natively is limiting though which is why I felt like tossing around this idea for something snappier than JSON/CSV but more flexible than buffer.
SL Feedback
updated the status to
tracked
SL Feedback
Issue tracked. We have no estimate when it may be implemented. Please see future updates here.