Hi Victor,
Indeed, the ability to limit the types permitted to be loaded
(contextually to what is unmarshalled) is important to minimize
unmarshalling-associated risks. In my work on Marshalling I currently
rely on an allow-list (essentially String -> Class, as I currently
find ClassLoader to have a rather large API surface for this purpose)
since parsing a schema descriptor is resolving types.
Transparent protocols allow for more inspection than opaque protocols,
the latter limits the allow list to classes. Our current wire protocol
allows deep introspection, but I haven't given this enough thought at
this stage as to how an API might look for a transparent protocol, vs
opaque. OIS relies on the context ClassLoader, or a stack walk when
attempting to resolve class types. A string allow list is simpler, but
you might also consider permitting the OIS to be given a specific
ClassLoader for type resolution, since these are different functions
where a ClassLoader can perform the function of the allow list, but
perhaps shouldn't, but still be responsible for class type resolution.
For schemaless payloads, the schema needs to be either known in
advance, or be determinable from the shape of the data itself.
I did consider this, and it really depends on the implementation, but
addressing transparency and deterministic behaviour is a simplification
that provides more certainty around security, favouring the developer,
or using inference which favours attackers by providing options, similar
to how a TLS protocol can be downgraded to a less secure version. I
didn't consider my capability sufficient to implement the latter.
Current designs of Marshalling are essentially T -> record -> output
and input -> record -> T where the record serves both as schema
definition (using the record components as both names and types are
present at runtime). This encoding also permits versioning
(structure-as-version) as well as translation (record-to-enum-value,
record-to-cache-lookup, or equivalent).
This has the benefit of type safety, and since records don't support
inheritance hierarchies, it's an elegant solution. We needed to
support existing inheritance hierarchies of Serializable objects, and we
had code that is compiled separately and comes together at runtime, so
we couldn't rely on the compiler checks for generic collections and
built collection type validators to address that.
I've been trialling AI since April this year. I've developed standards
I can't share on this list, since I've had AI agents assist with their
documentation, they're available for viewing on GitHub and anything
prior to April is AI free, and all AI contributions are documented in
git commits.
I've attached a text file that demonstrates the capabilities of the
serialisation protocol. I haven't invented a new protocol; it's DER
ASN.1, with canonical ordering rules for Collection interfaces that
allow for cross-language collection types.
JGDMS/JGDMS/examples/wire-protocol-showcase at trunk · pfirmstone/JGDMS
<https://github.com/pfirmstone/JGDMS/tree/trunk/JGDMS/examples/wire-protocol-showcase>
Note the adversarial test code that produced the attachment was written
with AI assistance, genuinely attacking it to find weaknesses in the
protocol.
>We don't think of the object's serial form as serialised fields; they
are serialised parameter arguments used to create new objects.
That resonates with how I view it as well.
:)
Cheers,
Peter.
Modules built and installed from current source.
########################################################################
# Building the showcase module (demos 1, 3, 4, 6)
########################################################################
########################################################################
# Demonstration 1: Same object, same bytes -- everywhere
########################################################################
========================================================================
Same object, same bytes -- everywhere
========================================================================
Source:
https://github.com/pfirmstone/JGDMS/blob/trunk/JGDMS/examples/wire-protocol-showcase/src/main/java/au/net/zeus/jgdms/showcase/demo/SameObjectSameBytesD
emo.java
Two separately built readings with the same value, in the canonical format:
pass 1 : 3064 3014 0203 0F42 6A0C 0D53 7461 7469 ...(102 bytes total)
pass 2 : 3064 3014 0203 0F42 6A0C 0D53 7461 7469 ...(102 bytes total)
identical bytes? true
checksum (SHA-256) of pass 1 :
b0c19e936efc9bf5c927ca58fc7528fe8b0967fee1413d8902637a791e90e9a4
checksum (SHA-256) of pass 2 :
b0c19e936efc9bf5c927ca58fc7528fe8b0967fee1413d8902637a791e90e9a4
same checksum? true
=> The checksum IS the identity of the value. Anyone, anywhere, who has
the same reading computes the same checksum, with no shared state.
A signature made over the canonical bytes, checked after the reading has
been written out and read back in:
bytes identical after the round trip? true
signature still verifies? true
Now two batches that are EQUAL by value:
batch A: the same reading object appears twice
batch B: two separate readings that are equal appear
are the two batches equal by value? true
canonical format : batch A = 1138 bytes, batch B = 1138 bytes -> identical?
true
Java built-in : batch A = 817 bytes, batch B = 857 bytes -> identical?
false
=> The canonical format gives equal values identical bytes. Java's built-in
serialization does not: it also records whether the two parts were the
same object, so equal values can come out as different bytes -- and a
checksum or signature over them would not match.
ALL CLAIMS HELD: canonical bytes are a stable, portable identity for a value.
(Running it a second time in a fresh process -- the checksum is identical,
because the bytes depend only on the value, not on the run.)
fresh process: checksum (SHA-256) of pass 1 :
b0c19e936efc9bf5c927ca58fc7528fe8b0967fee1413d8902637a791e90e9a4
########################################################################
# Demonstration 2: Match a template without ever loading the class
########################################################################
Building the shared-space library (one module, offline)...
Compiling the record class and the two programs...
Note:
C:\Users\peter\Documents\GitHub\JGDMS\JGDMS\examples\wire-protocol-showcase\demo2-match-without-the-class\src\au\net\zeus\jgdms\showcase\match\Server.java
uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
==================== PRODUCER (record class IS present) ====================
Producer (this program HAS the record class on its classpath):
record class .......... au.net.zeus.jgdms.showcase.entry.SensorReadingEntry
stored record ......... SensorReadingEntry{stationName=Station-North,
measuredQuantity=temperature, sequenceNumber=42}
matching template ..... SensorReadingEntry{stationName=null,
measuredQuantity=temperature, sequenceNumber=null} (null = match anything)
non-matching template . SensorReadingEntry{stationName=Station-South,
measuredQuantity=null, sequenceNumber=null}
wrote three on-the-wire records to:
C:/Users/peter/Documents/GitHub/JGDMS/JGDMS/examples/wire-protocol-showcase/demo2-match-without-the-class/out/records.wire
(each field is now canonical bytes; the class name is just text)
============ MATCHING SERVER (record class is NOT on the classpath) ========
server classpath deliberately EXCLUDES the folder with the record class.
Source:
https://github.com/pfirmstone/JGDMS/blob/trunk/JGDMS/examples/wire-protocol-showcase/demo2-match-without-the-class/src/au/net/zeus/jgdms/showcase/match
/Server.java
Server (this program does NOT have the record class on its classpath):
Can this server load au.net.zeus.jgdms.showcase.entry.SensorReadingEntry?
NO - ClassNotFoundException (the class is not here)
Read three on-the-wire records without loading the record class.
The server can see the class NAME as plain text:
"au.net.zeus.jgdms.showcase.entry.SensorReadingEntry"
...but it never turns that name into a loaded class to match.
matching template vs stored record -> matches? true (expected true)
non-matching template vs stored record -> matches? false (expected false)
If the server tried to rebuild the record OBJECT from the bytes:
UnusableEntryException (because UnmarshalException: Encountered a
ClassNotFoundException while unmarshalling
au.net.zeus.jgdms.showcase.entry.SensorReadin
gEntry; nested exception is:
java.lang.ClassNotFoundException:
au.net.zeus.jgdms.showcase.entry.SensorReadingEntry (no security manager: RMI
class loader disabled))
-> Rebuilding the object needs the class. MATCHING did not.
ALL CLAIMS HELD: the server matched templates by comparing bytes, with no
record class present, so there is no way for a hostile record to run code on
the serve
r during a match.
RESULT: demonstration succeeded (server exit code 0).
########################################################################
# Demonstration 3: The shape description travels once
########################################################################
========================================================================
The shape description travels in the stream -- and you pay for it once
========================================================================
Source:
https://github.com/pfirmstone/JGDMS/blob/trunk/JGDMS/examples/wire-protocol-showcase/src/main/java/au/net/zeus/jgdms/showcase/demo/SchemaSentOnceDemo.j
ava
We write 100 readings of the same type. Each reading's shape is a
family tree of three levels, with descriptive field names -- a realistic,
not-tiny shape description.
shape description (family tree) ...... 461 bytes
one self-describing record ........... 633 bytes (data + its own shape
description)
the shape is 73% of a single record
In the stream format, watch what each extra object adds:
first object (carries the shape) .......... 606 bytes so far
+ second object (refers to the shape) ..... +162 bytes
+ third object (refers to the shape) ..... +168 bytes
-> after the first, each object adds only its own data plus a short
back-reference.
Cumulative bytes to write N readings:
N | shape once | shape repeated | JSON w/ names
------+----------------+----------------+----------------
1 | 606 | 633 | 234
2 | 768 | 1258 | 461
5 | 1274 | 3153 | 1162
10 | 2104 | 6298 | 2317
20 | 3776 | 12600 | 4651
50 | 8784 | 31498 | 11671
100 | 17136 | 63000 | 23346
At N=100 (bar length = relative size):
shape once 17136 B #############
shape repeated 63000 B ################################################
JSON w/ names 23346 B ##################
Sending the shape once instead of every time saved 45,864 bytes (72.8%
smaller)
and it is 26.6% smaller than the equivalent JSON text (field names on every
record).
Why it matters: the reader never has to already know the shape. It arrives
in the stream, once, and every later object points back to it. You get a
self-describing stream without paying to describe every object.
Reported honestly: this win depends on the shape being a real fraction of
each object and shared across many objects. A prior measurement on a
different, flatter type (100 records, a tiny 125-byte shared shape) still
shrank 22,180 -> 9,378 bytes (to 42.3%). Where the shape is a bigger share,
as here, the saving is larger. Where each field is a separate, unique
capture (one storage layout in this project), the saving is only a few
percent -- and we would report that number, not this one.
########################################################################
# Demonstration 4: Feed it garbage, it stops politely
########################################################################
========================================================================
Feed it garbage, it stops politely
========================================================================
Source:
https://github.com/pfirmstone/JGDMS/blob/trunk/JGDMS/examples/wire-protocol-showcase/src/main/java/au/net/zeus/jgdms/showcase/demo/HostileInputDemo.jav
a
The reader that turns wire bytes back into objects is handed four hostile
inputs. It refuses each one cleanly -- a plain error, bounded memory, no
hang, no crash. It never rebuilds an object it has not first accepted, so a
refused message never gets the chance to run code.
(This whole demonstration is running inside a memory ceiling of about 128
megabytes.)
------------------------------------------------------------------------
1. A message cut off partway through
------------------------------------------------------------------------
a complete message of 606 bytes reads back fine: true
now the last 12 bytes are dropped, cutting it off partway through.
the length markers now promise more data than the message actually holds.
the reader refused it with a plain, checked error -- the ordinary kind a
program is expected to catch and handle.
=> HELD: the reader refused the cut-off message cleanly.
------------------------------------------------------------------------
2. A message whose structure markers have been scrambled
------------------------------------------------------------------------
the marker that says what kind of thing comes next is overwritten with
a value that means nothing to the reader.
the reader refused it with a plain, checked error -- the ordinary kind a
program is expected to catch and handle.
=> HELD: the reader refused the scrambled message cleanly.
------------------------------------------------------------------------
3. A message nested inside itself deeper than the reader will follow
------------------------------------------------------------------------
the reader follows nesting up to a fixed depth of 16 and no further.
a message nested exactly 16 deep reads back fine.
a message nested 5000 deep -- thousands of levels -- is offered.
(the crafted message is genuine, not malformed: its bytes are identical to
what the writer would produce -- confirmed here: true)
the reader stopped at its depth limit with a plain, checked error,
instead of following the nesting until it ran out of room.
=> HELD: the reader stopped at its depth limit instead of crashing.
------------------------------------------------------------------------
4. A tiny message crafted to balloon into gigabytes when unpacked
------------------------------------------------------------------------
The trick: send one large shape once, then a long run of tiny
back-references to it. Each back-reference is a few dozen bytes on the
wire, but each one would re-grow the whole large shape when unpacked, so a
small wire message would balloon into gigabytes in memory.
For a safe demonstration the reader is given a deliberately small input
budget, so its ceiling is small and it refuses after only a few dozen
re-growths -- without ever buffering the gigabytes a full-size budget
would otherwise permit.
Part A -- one hostile message (the memory bomb):
each 42-byte back-reference would re-grow to 65,514 bytes -- about 1560
times
more output for every byte on the wire.
the message on the wire: 107,544 bytes.
the reader's memory ceiling for one message: about 2 megabytes.
growth reached 2,096,000 bytes and then stopped, under that ceiling.
at the normal, full-size input budget the same trick would have reached
about 24 gigabytes -- refused there too, memory never following.
refused with a plain, checked error; memory stayed under the ceiling.
Part B -- a flood of small messages (the slow-burn version):
each message re-grows just one shape, small enough on its own -- but the
flood never stops. Memory for each message stays flat (the reader lets go
of one before starting the next), yet the reader still stops the flood
before the accumulated work runs away.
255 small messages were accepted, then the flood was refused.
memory for the last accepted message: 65,500 bytes -- it never grew across
the
whole flood; that is the bounded-memory guarantee.
refused with a plain, checked error; memory stayed flat throughout.
=> HELD: the reader refused both the single bomb and the flood, memory
bounded.
========================================================================
ALL FOUR HELD: every hostile input was refused cleanly, with memory kept in
bounds.
========================================================================
########################################################################
# Demonstration 5: Filter records by a written rule, without the class
########################################################################
Building the rule-language library and its wire-format dependencies (offline)...
Compiling the record classes and the two programs...
==================== PRODUCER (record classes ARE present) ====================
Producer (this program HAS the record classes on its classpath):
weather record class ....
au.net.zeus.jgdms.showcase.rule.records.WeatherReading
survey record class .....
au.net.zeus.jgdms.showcase.rule.records.SurveyObservation
wrote 5 weather readings:
WeatherReading{stationName='North-Ridge', temperatureCelsius=25.4,
humidityPercent=60.0, sequenceNumber=1}
WeatherReading{stationName='North-Vale', temperatureCelsius=14.2,
humidityPercent=82.0, sequenceNumber=2}
WeatherReading{stationName='South-Bay', temperatureCelsius=31.0,
humidityPercent=45.0, sequenceNumber=3}
WeatherReading{stationName='Northgate', temperatureCelsius=22.5,
humidityPercent=55.0, sequenceNumber=4}
WeatherReading{stationName='East-Field', temperatureCelsius=19.9,
humidityPercent=70.0, sequenceNumber=5}
wrote 1 survey observation:
SurveyObservation{targetName='Control-Point-A', bearingDegrees=30.0,
elevationDegrees=10.0, slopeDistanceMetres=100.0}
each field is now canonical bytes; the shape description travels alongside.
wrote everything to:
C:/Users/peter/Documents/GitHub/JGDMS/JGDMS/examples/wire-protocol-showcase/demo5-filter-by-a-rule/out/records.wire
============ READER (record classes are NOT on the classpath) =================
reader classpath deliberately EXCLUDES the folder with the record classes.
Source:
https://github.com/pfirmstone/JGDMS/blob/trunk/JGDMS/examples/wire-protocol-showcase/demo5-filter-by-a-rule/src/au/net/zeus/jgdms/showcase/rule/Reader.
java
Reader (this program does NOT have the record classes on its classpath):
[PASS] the weather record class is genuinely NOT loadable here
[PASS] the survey record class is genuinely NOT loadable here
Read 5 weather readings and 1 survey observation
straight into a class-free field view - no record object was built.
---------------------------------------------------------------------------
1. Filter readings by a written rule - without the record class
---------------------------------------------------------------------------
the rule (shown readably): (temperatureCelsius > 20.0 AND stationName
starts-with "North")
the rule travels as 76 canonical bytes; the reader never parsed text.
[PASS] the gate ACCEPTED the rule at registration (cost 12)
running the rule against each reading, through the class-free field view:
[WeatherReading] temp=25.4 name="North-Ridge" -> SELECT
[WeatherReading] temp=14.2 name="North-Vale" -> reject
[WeatherReading] temp=31.0 name="South-Bay" -> reject
[WeatherReading] temp=22.5 name="Northgate" -> SELECT
[WeatherReading] temp=19.9 name="East-Field" -> reject
[PASS] the rule selected exactly the warm northern readings, rejected the rest
[PASS] rebuilding a reading into its object would need the class; filtering
never did
---------------------------------------------------------------------------
2. The rule cannot misbehave - it is bounded by construction
---------------------------------------------------------------------------
[PASS] the language has exactly 27 fixed node kinds - none is a loop or
recursion, so no rule can iterate or call itself
[PASS] an over-expensive rule is REFUSED at registration (COST_EXCEEDED: cost
gate rejected: C(E) = 1572867 exceeds maxExprCost (1000000))
[PASS] an oversized rule (over the node ceiling) is REFUSED at registration
(DECODE_REJECTED: decode rejected: maxExprNodes (1024) exceeded at node 1025)
[PASS] a malformed rule is REFUSED cleanly, not crashed (DECODE_REJECTED:
decode rejected: Expected SEQUENCE (tag 0x30), got UNIVERSAL CONSTRUCTED 10)
[PASS] a rule reading a field that is not there fails closed with a definite
error (error ABSENT_FIELD)
---------------------------------------------------------------------------
3. The same rule, the same answer - exactly
---------------------------------------------------------------------------
[PASS] the yes/no rule gave the identical answer on all 1000 evaluations
the transform (shown readably):
east = slopeDistanceMetres * cos(radians(elevationDegrees)) *
sin(radians(bearingDegrees))
north = slopeDistanceMetres * cos(radians(elevationDegrees)) *
cos(radians(bearingDegrees))
up = slopeDistanceMetres * sin(radians(elevationDegrees))
[PASS] the gate ACCEPTED the transform at registration (cost 196)
result for "Control-Point-A" (bearing 30.0 deg, vertical angle 10.0 deg,
distance 100.0 m):
east = +49.240387651 m (exact bits 0x40489ec505c4ddf1)
north = +85.286853195 m (exact bits 0x4055525bcd8114f5)
up = +17.364817767 m (exact bits 0x40315d64b278f243)
[PASS] the transform gave the bit-for-bit identical vector on all 1000
evaluations
[PASS] the direction vector's length equals the slope distance (100.000000 m
vs 100.000000 m)
HONESTY: this shows one implementation giving the same answer every time.
The language is DESIGNED so a reader written in another language (Rust,
Haskell)
would give the byte-identical answer - pinned rounding, a canonical wire form,
and a shared corpus of conformance vectors that ships with the rule-language
module (evaluation, decode and registration vectors, including this
correctly-rounded trigonometry). That second-language reader is planned, not
yet running: nothing here claims a live cross-language comparison.
ALL CLAIMS HELD: a written rule filtered the readings without the record class,
the gate refused every rule that could misbehave, and the same rule gave the sam
e answer exactly - no downloaded code, no class needed, nothing that could hang.
RESULT: demonstration succeeded (reader exit code 0).
########################################################################
# Demonstration 6: Two different collection classes, one value, one encoding
########################################################################
========================================================================
Two different collection classes, one value, one encoding
========================================================================
Source:
https://github.com/pfirmstone/JGDMS/blob/trunk/JGDMS/examples/wire-protocol-showcase/src/main/java/au/net/zeus/jgdms/showcase/demo/CollectionEqualityDe
mo.java
The canonical format serialises a collection's VALUE (what .equals
compares), never the implementation class holding it. Stock Java
serialization (java.io.ObjectOutputStream) records the implementation
class too, so the same value comes out as different bytes from
different implementations.
Set field (declared java.util.Set -> canonical, octet-sorted):
the three records are equal by value (.equals)? true
canonical | HashSet : 49 bytes, SHA-256 ad9e246f6da1d5a7...
canonical | TreeSet : 49 bytes, SHA-256 ad9e246f6da1d5a7...
canonical | LinkedHashSet : 49 bytes, SHA-256 ad9e246f6da1d5a7...
-> canonical bytes identical across all three? true
stock Java | HashSet : 99 bytes, SHA-256 a9c38ea5565d2300...
stock Java | TreeSet : 92 bytes, SHA-256 4e31277fc54b8dfc...
stock Java | LinkedHashSet : 137 bytes, SHA-256 7178d5509f83982e...
-> stock Java serialization bytes all mutually DIFFERENT? true
List field (declared java.util.List -> order preserved; same order in both):
the two records are equal by value (.equals)? true
canonical | ArrayList : 46 bytes, SHA-256 8e476832081735bb...
canonical | LinkedList : 46 bytes, SHA-256 8e476832081735bb...
-> canonical bytes identical? true
stock Java | ArrayList : 101 bytes, SHA-256 9a053adcad702c7d...
stock Java | LinkedList : 91 bytes, SHA-256 0ea0c0038d2c122a...
-> stock Java serialization bytes DIFFERENT? true
Map field (declared java.util.Map -> canonical, entries sorted by encoded key):
the three records are equal by value (.equals)? true
canonical | HashMap : 70 bytes, SHA-256 7ccbf430d511f24d...
canonical | TreeMap : 70 bytes, SHA-256 7ccbf430d511f24d...
canonical | LinkedHashMap : 70 bytes, SHA-256 7ccbf430d511f24d...
-> canonical bytes identical across all three? true
stock Java | HashMap : 234 bytes, SHA-256 f20e0f2f886f8221...
stock Java | TreeMap : 236 bytes, SHA-256 9a43a144a799abfb...
stock Java | LinkedHashMap : 287 bytes, SHA-256 da9e64882a235b90...
-> stock Java serialization bytes all mutually DIFFERENT? true
The honest exception -- a field declared as LinkedHashSet itself:
LinkedHashSet GUARANTEES a deterministic (insertion) iteration order,
so the canonical format PRESERVES that order instead of sorting it.
Two routes with the same waypoints inserted in different orders:
equal by value (.equals)? true
canonical bytes identical? false
north-first: 49 bytes, SHA-256 7639bf98dca0dc23...
west-first : 49 bytes, SHA-256 f192f0a3ed15c2fb...
=> Intended, and documented in the wire-format spec: for insertion-
ordered types the insertion order is part of what you declared, so
byte-equality is deliberately STRICTER than .equals. Declare the
field as Set if only membership is the value.
The same rule across languages (informative -- see note below):
Java declared type | Wire form (discipline) | Rust
| Haskell
--------------------------+---------------------------------+-------------------------------+---------------------------------
Set / HashSet | SET OF, octet-sorted (canonical)| std HashSet<T>
| Data.HashSet (unordered-containers)
Map / HashMap | SET OF entries, key-sorted | std
HashMap<K,V> | Data.HashMap (unordered-containers)
List (ArrayList, ...) | SEQUENCE OF, order preserved | std
Vec<T>/VecDeque/LinkedList| [a], Data.Sequence.Seq (containers)
SortedSet / TreeSet | SEQUENCE OF, sorted-preserved | std BTreeSet<T>
| Data.Set (containers)
SortedMap / TreeMap | SEQUENCE OF, sorted-preserved | std
BTreeMap<K,V> | Data.Map (containers)
LinkedHashSet/-Map | SEQUENCE OF, insertion-preserved| indexmap
IndexSet/IndexMap | OSet/OMap (ordered-containers)
| | (crate --
none in std) | (package -- none in base)
PriorityQueue | SET OF, octet-sorted multiset | std
BinaryHeap<T> | MinQueue (pqueue) / Heap (heaps)
The insertion-ordered row carries the same honest exception shown above,
and it reproduces in Rust: indexmap's == ignores insertion order while its
iteration (and so its bytes) preserve it -- byte-equality stricter than ==.
Haskell's ordered-containers instead defines == order-SENSITIVELY, so there
bytes and == agree exactly. That the same fork appears, for the same reason,
in languages designed independently of this wire format is the evidence the
ordering rule is a property of collection semantics, not a Java quirk.
NOTE: the Rust and Haskell columns are the documented canonical-form mapping
from docs/der-rust-collection-mapping.md and docs/der-haskell-collection-
mapping.md -- research grounded in those languages' own documentation. The
Rust and Haskell encoders are future work, so this is NOT a live measured
cross-runtime byte comparison; the Java bytes above are the measured part.
ALL CLAIMS HELD: the wire encodes collection VALUES; the implementation
class never reaches the wire (and insertion order is kept exactly where
the declared type makes it part of the value).
########################################################################
# Demonstration 7: Filter records inside a live space, server-side, without the
class
########################################################################
Building the reactor modules (offline, tests skipped)...
Compiling the entry+client classes (out/entry) and the orchestrator (out/app)...
Note:
C:\Users\peter\Documents\GitHub\JGDMS\JGDMS\examples\wire-protocol-showcase\demo7-filter-pushdown\src\au\net\zeus\jgdms\showcase\pushdown\Rendezvous.java
uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Running demo7 (one JVM; the space's app loader OMITS WeatherReading, a child
loader over out/entry supplies it to the client logic)...
==================== demo7: class-free CEL filter pushdown ====================
[PASS] the space's class loader provably LACKS WeatherReading ->
java.lang.ClassNotFoundException:
au.net.zeus.jgdms.showcase.pushdown.records.WeatherReading
The space will filter temperatureCelsius > 20.0 &&
stationName.startsWith("North") class-free, over each candidate's
own DER v2 schema, without ever loading that class.
Starting transient Mahalo (transaction manager) ...
July 27, 2026 4:27:14 PM org.apache.river.mahalo.TxnManagerImpl$3 run
INFO: Mahalo started: org.apache.river.mahalo.TxnManagerImpl$3@45a4b042
Starting transient Outrigger (JavaSpace) ...
July 27, 2026 4:27:14 PM org.apache.river.outrigger.OutriggerServerImpl <init>
INFO: Outrigger server started:
org.apache.river.outrigger.OutriggerServerImpl@4278284b
Exported over plaintext TCP JERI (TcpServerEndpoint, integrity-only
constraints): Outrigger=AtomicDerILFactory, Mahalo=AtomicILFactory; no TLS.
client (child loader) can load WeatherReading and authored the filter:
temperatureCelsius > 20.0 && stationName.startsWith("North") (83 canonical
FilterEnvelope bytes)
filtered contents returned: [22.0|Northgate, 25.0|North Ridge]
[PASS] §8.2 filtered contents = exactly the warm-northern entries
[22.0|Northgate, 25.0|North Ridge]
[PASS] §8.4b a warm-northern NorthStationReading (subclass) was returned,
filtered by inherited fields
filtered take returned: [22.0|Northgate, 25.0|North Ridge]
[PASS] §8.2 filtered take removed exactly the warm-northern entries
[22.0|Northgate, 25.0|North Ridge]
[PASS] §8.4a a cold-northern write did NOT satisfy the filtered read
(evaluated false, not fail-open)
[PASS] §8.4a the blocking filtered read resolved to the later warm-northern
write (WeatherReading{temperatureCelsius=26.0, stationName=North Star})
---- §8.2 class-free filtering (server-side counter deltas) ----
filter.evaluated += 12, filter.passed += 5, filter.excludedFalse += 7,
filter.failClosedExclusions += 0
[PASS] §8.2 candidates were EVALUATED and cleanly rejected (evaluated >
passed)
[PASS] §8.2 at least one candidate PASSED the predicate
[PASS] §8.2 happy path: ZERO fail-closed exclusions (every candidate decoded)
[PASS] §8.2/§8.4a/b client-side identity checks all held
[PASS] §8.4c the >64 KiB warm-northern candidate was EXCLUDED (budget);
targeted filtered read returned null
---- §8.4c projection-budget bound (>64 KiB candidate) ----
filter.rejected.projectionBudget delta = 1, a PROJECTION_BUDGET
FilterEvaluation event appeared = true
[PASS] §8.4c the >64 KiB candidate was EXCLUDED and counted in
filter.rejected.projectionBudget
[PASS] §8.4c a PROJECTION_BUDGET FilterEvaluation event was recorded
[PASS] §8.4c client's targeted filtered read on the oversized candidate
returned null
---- §8.3 confused-deputy (INV-1), best effort (non-gating) ----
[LIMITATION] the confused-deputy end-to-end flow could not be demonstrated in
this minimal, no-download environment.
§8.3 marshals two exported proxies to the space -- P's
RemoteEventListener stub and Mahalo's
transaction-manager proxy (embedded in the ServerTransaction) --
which must be reconstructed at the
Outrigger endpoint via a bootstrap proxy resolved through an
integrity-verified codebase (a lookup
service / httpmd class server) that this demo omits. This is
TRANSPORT-INDEPENDENT (identical over
tcp and ssl) and orthogonal to the CEL filter. INV-1 is covered
by SiteFStructuralTest + the watcher
unit tests + the server-side observation above; NOT a filter
defect, does not affect the result below.
root cause: java.rmi.UnmarshalException: exception unmarshalling
response; nested exception is:
java.io.IOException: DER object stream: cannot read the stream-format
version octet (STD-006 Appendix C sec.C.5.2): Unexpected end of DER input: need
1
byte(s) at pos=0, end=0
---- final operator counter snapshot ----
filter.evaluated = 13
filter.passed = 5
filter.excludedFalse = 7
filter.failClosedExclusions = 1
filter.rejected.projectionBudget = 1
---- FilterEvaluation JFR events by outcome ----
PASSED=5 EXCLUDED_FALSE=7 FAIL_CLOSED=0 PROJECTION_BUDGET=1 other=0
(total=13)
RESULT: demo7 succeeded (class-free, value-expressive, fail-closed; §8.3
attempted, see note).
exit code = 0
RESULT: demo7 succeeded (exit code 0).
########################################################################
# Automated checks (mvn test)
########################################################################
========================================================================
Feed it garbage, it stops politely
========================================================================
Source:
https://github.com/pfirmstone/JGDMS/blob/trunk/JGDMS/examples/wire-protocol-showcase/src/main/java/au/net/zeus/jgdms/showcase/demo/HostileInputDemo.jav
a
The reader that turns wire bytes back into objects is handed four hostile
inputs. It refuses each one cleanly -- a plain error, bounded memory, no
hang, no crash. It never rebuilds an object it has not first accepted, so a
refused message never gets the chance to run code.
(This whole demonstration is running inside a memory ceiling of about 16328
megabytes.)
------------------------------------------------------------------------
1. A message cut off partway through
------------------------------------------------------------------------
a complete message of 606 bytes reads back fine: true
now the last 12 bytes are dropped, cutting it off partway through.
the length markers now promise more data than the message actually holds.
the reader refused it with a plain, checked error -- the ordinary kind a
program is expected to catch and handle.
=> HELD: the reader refused the cut-off message cleanly.
------------------------------------------------------------------------
2. A message whose structure markers have been scrambled
------------------------------------------------------------------------
the marker that says what kind of thing comes next is overwritten with
a value that means nothing to the reader.
the reader refused it with a plain, checked error -- the ordinary kind a
program is expected to catch and handle.
=> HELD: the reader refused the scrambled message cleanly.
------------------------------------------------------------------------
3. A message nested inside itself deeper than the reader will follow
------------------------------------------------------------------------
the reader follows nesting up to a fixed depth of 16 and no further.
a message nested exactly 16 deep reads back fine.
a message nested 5000 deep -- thousands of levels -- is offered.
(the crafted message is genuine, not malformed: its bytes are identical to
what the writer would produce -- confirmed here: true)
the reader stopped at its depth limit with a plain, checked error,
instead of following the nesting until it ran out of room.
=> HELD: the reader stopped at its depth limit instead of crashing.
------------------------------------------------------------------------
4. A tiny message crafted to balloon into gigabytes when unpacked
------------------------------------------------------------------------
The trick: send one large shape once, then a long run of tiny
back-references to it. Each back-reference is a few dozen bytes on the
wire, but each one would re-grow the whole large shape when unpacked, so a
small wire message would balloon into gigabytes in memory.
For a safe demonstration the reader is given a deliberately small input
budget, so its ceiling is small and it refuses after only a few dozen
re-growths -- without ever buffering the gigabytes a full-size budget
would otherwise permit.
Part A -- one hostile message (the memory bomb):
each 42-byte back-reference would re-grow to 65,514 bytes -- about 1560
times
more output for every byte on the wire.
the message on the wire: 107,544 bytes.
the reader's memory ceiling for one message: about 2 megabytes.
growth reached 2,096,000 bytes and then stopped, under that ceiling.
at the normal, full-size input budget the same trick would have reached
about 24 gigabytes -- refused there too, memory never following.
refused with a plain, checked error; memory stayed under the ceiling.
Part B -- a flood of small messages (the slow-burn version):
each message re-grows just one shape, small enough on its own -- but the
flood never stops. Memory for each message stays flat (the reader lets go
of one before starting the next), yet the reader still stops the flood
before the accumulated work runs away.
255 small messages were accepted, then the flood was refused.
memory for the last accepted message: 65,500 bytes -- it never grew across
the
whole flood; that is the bounded-memory guarantee.
refused with a plain, checked error; memory stayed flat throughout.
=> HELD: the reader refused both the single bomb and the flood, memory
bounded.
========================================================================
ALL FOUR HELD: every hostile input was refused cleanly, with memory kept in
bounds.
========================================================================
========================================================================
Same object, same bytes -- everywhere
========================================================================
Source:
https://github.com/pfirmstone/JGDMS/blob/trunk/JGDMS/examples/wire-protocol-showcase/src/main/java/au/net/zeus/jgdms/showcase/demo/SameObjectSameBytesD
emo.java
Two separately built readings with the same value, in the canonical format:
pass 1 : 3064 3014 0203 0F42 6A0C 0D53 7461 7469 ...(102 bytes total)
pass 2 : 3064 3014 0203 0F42 6A0C 0D53 7461 7469 ...(102 bytes total)
identical bytes? true
checksum (SHA-256) of pass 1 :
b0c19e936efc9bf5c927ca58fc7528fe8b0967fee1413d8902637a791e90e9a4
checksum (SHA-256) of pass 2 :
b0c19e936efc9bf5c927ca58fc7528fe8b0967fee1413d8902637a791e90e9a4
same checksum? true
=> The checksum IS the identity of the value. Anyone, anywhere, who has
the same reading computes the same checksum, with no shared state.
A signature made over the canonical bytes, checked after the reading has
been written out and read back in:
bytes identical after the round trip? true
signature still verifies? true
Now two batches that are EQUAL by value:
batch A: the same reading object appears twice
batch B: two separate readings that are equal appear
are the two batches equal by value? true
canonical format : batch A = 1138 bytes, batch B = 1138 bytes -> identical?
true
Java built-in : batch A = 817 bytes, batch B = 857 bytes -> identical?
false
=> The canonical format gives equal values identical bytes. Java's built-in
serialization does not: it also records whether the two parts were the
same object, so equal values can come out as different bytes -- and a
checksum or signature over them would not match.
ALL CLAIMS HELD: canonical bytes are a stable, portable identity for a value.
========================================================================
The shape description travels in the stream -- and you pay for it once
========================================================================
Source:
https://github.com/pfirmstone/JGDMS/blob/trunk/JGDMS/examples/wire-protocol-showcase/src/main/java/au/net/zeus/jgdms/showcase/demo/SchemaSentOnceDemo.j
ava
We write 100 readings of the same type. Each reading's shape is a
family tree of three levels, with descriptive field names -- a realistic,
not-tiny shape description.
shape description (family tree) ...... 461 bytes
one self-describing record ........... 633 bytes (data + its own shape
description)
the shape is 73% of a single record
In the stream format, watch what each extra object adds:
first object (carries the shape) .......... 606 bytes so far
+ second object (refers to the shape) ..... +162 bytes
+ third object (refers to the shape) ..... +168 bytes
-> after the first, each object adds only its own data plus a short
back-reference.
Cumulative bytes to write N readings:
N | shape once | shape repeated | JSON w/ names
------+----------------+----------------+----------------
1 | 606 | 633 | 234
2 | 768 | 1258 | 461
5 | 1274 | 3153 | 1162
10 | 2104 | 6298 | 2317
20 | 3776 | 12600 | 4651
50 | 8784 | 31498 | 11671
100 | 17136 | 63000 | 23346
At N=100 (bar length = relative size):
shape once 17136 B #############
shape repeated 63000 B ################################################
JSON w/ names 23346 B ##################
Sending the shape once instead of every time saved 45,864 bytes (72.8%
smaller)
and it is 26.6% smaller than the equivalent JSON text (field names on every
record).
Why it matters: the reader never has to already know the shape. It arrives
in the stream, once, and every later object points back to it. You get a
self-describing stream without paying to describe every object.
Reported honestly: this win depends on the shape being a real fraction of
each object and shared across many objects. A prior measurement on a
different, flatter type (100 records, a tiny 125-byte shared shape) still
shrank 22,180 -> 9,378 bytes (to 42.3%). Where the shape is a bigger share,
as here, the saving is larger. Where each field is a separate, unique
capture (one storage layout in this project), the saving is only a few
percent -- and we would report that number, not this one.
########################################################################
# Summary
########################################################################
[PASS] Demo 1: Same object, same bytes
[PASS] Demo 2: Match without the class
[PASS] Demo 3: Schema sent once
[PASS] Demo 4: Feed it garbage
[PASS] Demo 5: Filter by a rule
[PASS] Demo 6: Collection equality
[PASS] Demo 7: Filter pushdown into a live space
[PASS] Automated checks (mvn test)
RESULT: all demonstrations and automated checks passed.
