10 Commits
Author SHA1 Message Date
weaselbot 717f30099f schemagen: cache non-object $defs entries to avoid duplicate types
Extend the existing per-definition cache (`self._building`) to enum,
array, and scalar $defs, not just object definitions. This ensures
that multiple $refs to the same non-object definition reuse the same
C++ type instead of generating Role, Role2, Role3, etc.

- Cache the built (type, nullable) tuple under defname for enum,
  scalar, and array definitions.
- Pre-register array definitions before recursing into items so $ref
  cycles resolve to the same TArr instance.
- Store object definitions as (TObj, nullable) tuples so nullable object
  $defs also preserve their nullability when referenced.

Add a regression test for issue #17 covering reused enum and array-of-enum
$defs.
2026-06-23 22:16:34 -04:00
andrew 1404ebdfbd Merge pull request 'fix schemagen integer parsing boundary bugs' (#30) from weaselbot/weaseljson:weaselbot/issue-19 into main
Reviewed-on: weaselab/weaseljson#30
2026-06-24 00:40:09 +00:00
weaselbot 38079cc278 schemagen: parse integer fallback exactly for decimal/exponent forms
Replace the double-based fallback in generated integer slots with a
string-to-int64 parser that handles decimal points and exponents
without losing precision near the int64 boundaries.

The old path used std::from_chars<double> and compared against
±9223372036854775808.0, which rounds the int64 max and min so that
valid values are rejected and out-of-range negatives are accepted.

The new helper:
- Parses sign, integer part, optional fraction, and optional exponent.
- Strips trailing zeros to cancel fractional places.
- Rejects non-integral values and overflow using exact uint64_t
  arithmetic.

Adds regression tests covering root integer and object-field integer
boundary values, including the cases from issue #19.
2026-06-23 18:36:53 -04:00
andrew ceb4bc1041 Merge pull request 'schemagen: drop support for additionalProperties: true' (#28) from weaselbot/weaseljson:weaselbot/issue-20 into main
Reviewed-on: weaselab/weaseljson#28
2026-06-23 20:56:58 +00:00
weaselbot 16f13c241c schemagen: drop support for additionalProperties: true
Following review feedback, the generator no longer supports permissive
objects. Changes:

- Reject `additionalProperties: true` at generation time.
- Treat an absent `additionalProperties` as `false`, so every object is
  strict by default and unknown keys are rejected during parsing.
- Remove the now-dead permissive-object infrastructure: `Kind::Skip`,
  `Cat::Skip`, `kSkip`, the per-frame `unknown` key set, and `isStrict()`.
- Update the README feature/rejection tables accordingly.
- Remove the permissive "loose" object from example.schema.json and the
  associated tests from test_gen.cpp.
- Add Python unit tests verifying the new `additionalProperties` behavior.

All tests pass (`ctest --output-on-failure`).
2026-06-23 15:11:48 -04:00
weaselbot ab95fefb09 schemagen: reject duplicate unknown keys in non-strict objects
Track unknown keys in a per-object unordered_set so that permissive
objects (additionalProperties absent/true) still reject duplicate keys,
matching the README guarantee.

- Add std::unordered_set<std::string> to Frame.
- Insert unknown keys in cbKeyData and reject duplicates before skipping.
- Add a permissive "loose" subobject to example.schema.json.
- Test single unknown key accepted and duplicate unknown/known keys rejected.
2026-06-23 15:08:48 -04:00
andrew e8830e27e9 Merge pull request 'schemagen: reserve generated Root/Skip/RootScalar and avoid Kind/ArrN collisions' (#27) from weaselbot/weaseljson:weaselbot/issue-21 into main
Reviewed-on: weaselab/weaseljson#27
2026-06-23 17:48:57 +00:00
andrew 5427db4b3d Merge pull request 'schemagen: fix nullable root types (#13)' (#29) from weaselbot/weaseljson:weaselbot/issue-13 into main
Reviewed-on: weaselab/weaseljson#29
2026-06-23 17:43:19 +00:00
weaselbot b5491afb38 schemagen: fix nullable root types (#13)
The C++ code generator now handles schemas where the top-level type is
nullable ("type": ["object", "null"], ["string", "null"], or
["array", "null"]).

Changes to weaseljson_schemagen.py:
- Rename the inner object struct when the root is a nullable object, so
  the `using Root = std::optional<...>` alias no longer conflicts with
  `struct Root`.
- Treat nullable root objects and arrays as container roots, emplacing
  the inner value before pushing the root frame and pointing the frame at
  the contained value.
- For nullable root scalars/enums, engage() now returns a pointer to the
  value inside the optional rather than to the optional wrapper itself.
- cbNull() now safely accepts a top-level null when the root is nullable
  and rejects it otherwise.

Regression tests added:
- nullable_object.schema.json + test_nullable_root.cpp
- nullable_string.schema.json
- nullable_array.schema.json

Closes #13
2026-06-23 12:58:28 -04:00
weaselbot 46ff8e2164 schemagen: reserve generated Root/Skip/RootScalar and avoid Kind/ArrN collisions
Make the type-name allocator aware of the identifiers the generator emits
itself (`Root` alias, `Skip`/`RootScalar` Kind enumerators) so user `$defs`
names can no longer collide with them.  Array-kind names (`Arr0`, `Arr1`, ...)
are now allocated only after checking for object/enum names, preventing
duplicate `Kind` enumerators when a schema defines e.g. `Arr0`.

Add Python regression tests that also syntax-check the generated headers
with a C++ compiler.

Closes #21
2026-06-23 12:23:09 -04:00
8 changed files with 805 additions and 83 deletions
+48
View File
@@ -46,6 +46,49 @@ target_link_libraries(schemagen_big PRIVATE ${PROJECT_NAME})
target_compile_options(schemagen_big PRIVATE -Wno-switch-enum)
add_dependencies(schemagen_big schemagen_big_h)
set(NULLABLE_OBJECT_SCHEMA
${CMAKE_CURRENT_SOURCE_DIR}/nullable_object.schema.json)
set(NULLABLE_STRING_SCHEMA
${CMAKE_CURRENT_SOURCE_DIR}/nullable_string.schema.json)
set(NULLABLE_ARRAY_SCHEMA
${CMAKE_CURRENT_SOURCE_DIR}/nullable_array.schema.json)
set(NULLABLE_OBJECT_H ${CMAKE_CURRENT_BINARY_DIR}/nullable_object.h)
set(NULLABLE_STRING_H ${CMAKE_CURRENT_BINARY_DIR}/nullable_string.h)
set(NULLABLE_ARRAY_H ${CMAKE_CURRENT_BINARY_DIR}/nullable_array.h)
add_custom_command(
OUTPUT ${NULLABLE_OBJECT_H}
COMMAND ${Python3_EXECUTABLE} ${SCHEMAGEN_SCRIPT} ${NULLABLE_OBJECT_SCHEMA} -o
${NULLABLE_OBJECT_H} --namespace nullable_object
DEPENDS ${SCHEMAGEN_SCRIPT} ${NULLABLE_OBJECT_SCHEMA}
COMMENT "Generating nullable_object.h")
add_custom_command(
OUTPUT ${NULLABLE_STRING_H}
COMMAND ${Python3_EXECUTABLE} ${SCHEMAGEN_SCRIPT} ${NULLABLE_STRING_SCHEMA} -o
${NULLABLE_STRING_H} --namespace nullable_string
DEPENDS ${SCHEMAGEN_SCRIPT} ${NULLABLE_STRING_SCHEMA}
COMMENT "Generating nullable_string.h")
add_custom_command(
OUTPUT ${NULLABLE_ARRAY_H}
COMMAND ${Python3_EXECUTABLE} ${SCHEMAGEN_SCRIPT} ${NULLABLE_ARRAY_SCHEMA} -o
${NULLABLE_ARRAY_H} --namespace nullable_array
DEPENDS ${SCHEMAGEN_SCRIPT} ${NULLABLE_ARRAY_SCHEMA}
COMMENT "Generating nullable_array.h")
add_custom_target(
schemagen_nullable_h DEPENDS ${NULLABLE_OBJECT_H} ${NULLABLE_STRING_H}
${NULLABLE_ARRAY_H})
add_executable(schemagen_nullable_root
${CMAKE_CURRENT_SOURCE_DIR}/test_nullable_root.cpp)
target_include_directories(schemagen_nullable_root
PRIVATE include ${CMAKE_CURRENT_BINARY_DIR})
target_link_libraries(schemagen_nullable_root PRIVATE ${PROJECT_NAME})
target_compile_options(schemagen_nullable_root PRIVATE -Wno-switch-enum)
add_dependencies(schemagen_nullable_root schemagen_nullable_h)
add_test(
NAME schemagen_example
COMMAND schemagen_example
@@ -55,3 +98,8 @@ add_test(
NAME schemagen_big
COMMAND schemagen_big
WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR})
add_test(
NAME schemagen_nullable_root
COMMAND schemagen_nullable_root
WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR})
+5 -5
View File
@@ -45,7 +45,7 @@ into the result, so it is non-movable.
| `$ref` to `$defs`/`definitions` | the referenced named struct |
| recursive `$ref` | `std::unique_ptr<T>` (cycle broken) |
| `additionalProperties: false` | unknown keys rejected |
| `additionalProperties` absent / `true` | unknown keys' values skipped |
| `additionalProperties` absent / `true` | not supported (rejected at generation) |
## Schema violations (rejected at parse time)
@@ -55,15 +55,15 @@ into the result, so it is non-movable.
- value not in a string `enum`
- a number not representable in the target type (e.g. `1.5` for an `integer`)
- duplicate object keys
- unknown key under `additionalProperties: false`
- unknown key in any object
## Not supported (rejected at generation time, no fallback)
`oneOf` / `anyOf` / `allOf` / `not` / `if`-`then`-`else`,
`patternProperties`, `additionalProperties` with a schema (typed map),
`prefixItems` (tuples), `const`, `dependentSchemas`/`dependentRequired`,
union `type` lists other than `["T", "null"]`, non-string enums, and remote
(`$ref` to other documents).
`additionalProperties: true`, `prefixItems` (tuples), `const`,
`dependentSchemas`/`dependentRequired`, union `type` lists other than
`["T", "null"]`, non-string enums, and remote (`$ref` to other documents).
## Notes
@@ -0,0 +1,9 @@
{
"type": [
"array",
"null"
],
"items": {
"type": "integer"
}
}
@@ -0,0 +1,15 @@
{
"type": [
"object",
"null"
],
"additionalProperties": false,
"required": [
"x"
],
"properties": {
"x": {
"type": "string"
}
}
}
@@ -0,0 +1,6 @@
{
"type": [
"string",
"null"
]
}
+158
View File
@@ -0,0 +1,158 @@
// Regression test for issue #13: nullable root types.
#include <cassert>
#include <cstdio>
#include <string>
#include "nullable_array.h"
#include "nullable_object.h"
#include "nullable_string.h"
static int failures = 0;
#define CHECK(cond) \
do { \
if (!(cond)) { \
printf("FAIL %s:%d: %s\n", __FILE__, __LINE__, #cond); \
++failures; \
} \
} while (0)
static WeaselJsonStatus parseStrided(nullable_object::RootBuilder &b,
std::string in) {
for (size_t i = 0; i < in.size(); ++i) {
char c = in[i];
WeaselJsonStatus s = b.feed(&c, 1);
if (s != WeaselJson_AGAIN)
return s;
}
return b.finish();
}
static WeaselJsonStatus parseStrided(nullable_string::RootBuilder &b,
std::string in) {
for (size_t i = 0; i < in.size(); ++i) {
char c = in[i];
WeaselJsonStatus s = b.feed(&c, 1);
if (s != WeaselJson_AGAIN)
return s;
}
return b.finish();
}
static WeaselJsonStatus parseStrided(nullable_array::RootBuilder &b,
std::string in) {
for (size_t i = 0; i < in.size(); ++i) {
char c = in[i];
WeaselJsonStatus s = b.feed(&c, 1);
if (s != WeaselJson_AGAIN)
return s;
}
return b.finish();
}
static void expectReject(nullable_object::RootBuilder &b, std::string in,
const char *what) {
WeaselJsonStatus s = parseStrided(b, in);
if (s == WeaselJson_REJECT) {
printf("ok reject: %s\n", what);
} else {
printf("FAIL expected reject (%s) got status %d for: %s\n", what, s,
in.c_str());
++failures;
}
}
int main() {
// ---- nullable root object: valid document ----
{
nullable_object::RootBuilder b;
WeaselJsonStatus s = parseStrided(b, R"({"x":"hello"})");
CHECK(s == WeaselJson_OK);
if (s == WeaselJson_OK) {
nullable_object::Root r = b.take();
CHECK(r.has_value());
CHECK(r->x == "hello");
printf("ok nullable root object accepts object\n");
}
}
// ---- nullable root object: null document ----
{
nullable_object::RootBuilder b;
WeaselJsonStatus s = parseStrided(b, "null");
CHECK(s == WeaselJson_OK);
if (s == WeaselJson_OK) {
nullable_object::Root r = b.take();
CHECK(!r.has_value());
printf("ok nullable root object accepts null\n");
}
}
// ---- nullable root object: schema checks still run ----
{
nullable_object::RootBuilder b;
expectReject(b, R"({"x":"hello","extra":1})",
"unknown key in strict nullable root object");
}
{
nullable_object::RootBuilder b;
expectReject(b, R"({})", "missing required field in nullable root object");
}
// ---- nullable root string: valid value ----
{
nullable_string::RootBuilder b;
WeaselJsonStatus s = parseStrided(b, R"("hello")");
CHECK(s == WeaselJson_OK);
if (s == WeaselJson_OK) {
nullable_string::Root r = b.take();
CHECK(r.has_value());
CHECK(*r == "hello");
printf("ok nullable root string accepts string\n");
}
}
// ---- nullable root string: null value ----
{
nullable_string::RootBuilder b;
WeaselJsonStatus s = parseStrided(b, "null");
CHECK(s == WeaselJson_OK);
if (s == WeaselJson_OK) {
nullable_string::Root r = b.take();
CHECK(!r.has_value());
printf("ok nullable root string accepts null\n");
}
}
// ---- nullable root array: valid value ----
{
nullable_array::RootBuilder b;
WeaselJsonStatus s = parseStrided(b, "[1,2,3]");
CHECK(s == WeaselJson_OK);
if (s == WeaselJson_OK) {
nullable_array::Root r = b.take();
CHECK(r.has_value());
CHECK(r->size() == 3);
CHECK((*r)[0] == 1 && (*r)[1] == 2 && (*r)[2] == 3);
printf("ok nullable root array accepts array\n");
}
}
// ---- nullable root array: null value ----
{
nullable_array::RootBuilder b;
WeaselJsonStatus s = parseStrided(b, "null");
CHECK(s == WeaselJson_OK);
if (s == WeaselJson_OK) {
nullable_array::Root r = b.take();
CHECK(!r.has_value());
printf("ok nullable root array accepts null\n");
}
}
if (failures == 0) {
printf("\nALL TESTS PASSED\n");
return 0;
}
printf("\n%d FAILURE(S)\n", failures);
return 1;
}
+382
View File
@@ -3,9 +3,12 @@
import json
import os
import re
import shutil
import subprocess
import sys
import tempfile
import textwrap
import unittest
SCRIPT = os.path.join(os.path.dirname(__file__), "weaseljson_schemagen.py")
@@ -87,5 +90,384 @@ class SchemagenKeywordTest(unittest.TestCase):
self.assertNotIn(f"std::optional<std::string> {kw};", stdout)
class SchemagenCollisionTest(unittest.TestCase):
"""Regression tests for issue #21: generated Root alias / Kind enum collisions."""
def setUp(self):
self.repo_root = os.path.dirname(
os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
)
self.include_dir = os.path.join(self.repo_root, "include")
self.compiler = shutil.which("c++")
def generate_and_compile(self, schema):
"""Run schemagen on schema and syntax-check the resulting header."""
with tempfile.TemporaryDirectory() as tmpdir:
schema_path = os.path.join(tmpdir, "schema.json")
with open(schema_path, "w") as fp:
json.dump(schema, fp)
header_path = os.path.join(tmpdir, "gen.h")
cmd = [
sys.executable,
SCRIPT,
schema_path,
"-o",
header_path,
"--namespace",
"test_schema",
]
result = subprocess.run(cmd, capture_output=True, text=True, check=False)
self.assertEqual(result.returncode, 0, msg=result.stderr)
if self.compiler:
cpp_path = os.path.join(tmpdir, "test.cpp")
with open(cpp_path, "w") as fp:
fp.write(
'#include "gen.h"\n'
"int main() {\n"
" test_schema::RootBuilder b;\n"
" test_schema::Root r = b.take();\n"
" (void)r;\n"
"}\n"
)
comp = subprocess.run(
[
self.compiler,
"-std=c++20",
"-fsyntax-only",
"-I",
self.include_dir,
"-I",
tmpdir,
cpp_path,
],
capture_output=True,
text=True,
check=False,
)
self.assertEqual(comp.returncode, 0, msg=comp.stderr)
with open(header_path) as fp:
return fp.read()
def test_root_alias_does_not_collide_with_user_type(self):
schema = {
"type": "array",
"items": {"$ref": "#/$defs/Root"},
"$defs": {"Root": {"enum": ["a", "b"]}},
}
out = self.generate_and_compile(schema)
self.assertIn("enum class Root1 : int { a, b };", out)
self.assertIn("using Root = std::vector<Root1>;", out)
self.assertNotIn("using Root = std::vector<Root>;", out)
def test_kind_enum_does_not_duplicate_arr0(self):
schema = {
"type": "object",
"properties": {
"arr": {"type": "array", "items": {"type": "string"}},
"obj": {"$ref": "#/$defs/Arr0"},
},
"$defs": {"Arr0": {"type": "object", "properties": {}}},
}
out = self.generate_and_compile(schema)
self.assertIn("struct Arr0", out)
m = re.search(r"enum class Kind : uint8_t \{([^}]+)\}", out)
self.assertIsNotNone(m)
enumerators = [e.strip() for e in m.group(1).split(",")]
self.assertIn("Arr0", enumerators)
self.assertIn("Arr1", enumerators)
self.assertEqual(len(enumerators), len(set(enumerators)))
def test_skip_user_type_is_allowed(self):
schema = {
"type": "array",
"items": {"$ref": "#/$defs/Skip"},
"$defs": {"Skip": {"type": "object", "properties": {}}},
}
out = self.generate_and_compile(schema)
self.assertIn("struct Skip", out)
self.assertNotIn("struct Skip1", out)
m = re.search(r"enum class Kind : uint8_t \{([^}]+)\}", out)
self.assertIsNotNone(m)
enumerators = [e.strip() for e in m.group(1).split(",")]
self.assertIn("Skip", enumerators)
self.assertIn("Arr0", enumerators)
self.assertEqual(len(enumerators), len(set(enumerators)))
def test_non_object_defs_reused_across_refs(self):
"""Regression test for issue #17: enum and array $defs must be reused."""
schema = {
"type": "object",
"properties": {
"role1": {"$ref": "#/$defs/Role"},
"role2": {"$ref": "#/$defs/Role"},
"roles1": {"$ref": "#/$defs/Roles"},
"roles2": {"$ref": "#/$defs/Roles"},
},
"$defs": {
"Role": {"enum": ["admin", "user"]},
"Roles": {"type": "array", "items": {"$ref": "#/$defs/Role"}},
},
}
out = self.generate_and_compile(schema)
# Exactly one Role enum is generated.
self.assertIn("enum class Role : int { admin, user };", out)
self.assertNotIn("enum class Role1 : int", out)
self.assertNotIn("enum class Role2 : int", out)
# The array of enum is represented by a single kind.
m = re.search(r"enum class Kind : uint8_t \{([^}]+)\}", out)
self.assertIsNotNone(m)
enumerators = [e.strip() for e in m.group(1).split(",")]
self.assertEqual(enumerators.count("Arr0"), 1)
# All four fields use the same C++ types.
self.assertIn("std::optional<Role> role1;", out)
self.assertIn("std::optional<Role> role2;", out)
self.assertIn("std::optional<std::vector<Role>> roles1;", out)
self.assertIn("std::optional<std::vector<Role>> roles2;", out)
class SchemagenAdditionalPropertiesTest(unittest.TestCase):
def run_schemagen(self, schema, args=None):
"""Run schemagen on a schema dict. Returns (returncode, stdout, stderr)."""
with tempfile.NamedTemporaryFile("w", suffix=".json", delete=False) as fp:
json.dump(schema, fp)
schema_path = fp.name
try:
cmd = [sys.executable, SCRIPT, schema_path]
if args:
cmd.extend(args)
result = subprocess.run(cmd, capture_output=True, text=True, check=False)
return result.returncode, result.stdout, result.stderr
finally:
os.unlink(schema_path)
def test_additional_properties_true_rejected(self):
schema = {
"type": "object",
"additionalProperties": True,
"properties": {"name": {"type": "string"}},
}
rc, stdout, stderr = self.run_schemagen(schema)
self.assertNotEqual(rc, 0)
self.assertIn("additionalProperties: true is not supported", stderr)
def test_additional_properties_absent_defaults_to_strict(self):
schema = {
"type": "object",
"properties": {"name": {"type": "string"}},
}
rc, stdout, stderr = self.run_schemagen(schema)
self.assertEqual(rc, 0, msg=stderr)
# The generated parser should reject unknown keys. Verify the key-matching
# helper returns -1 for an unknown key and cbKeyData rejects it.
self.assertIn("int matchKey(Kind k, std::string_view key) const {", stdout)
self.assertNotIn("bool isStrict(Kind k) const", stdout)
def test_additional_properties_false_accepted(self):
schema = {
"type": "object",
"additionalProperties": False,
"properties": {"name": {"type": "string"}},
}
rc, stdout, stderr = self.run_schemagen(schema)
self.assertEqual(rc, 0, msg=stderr)
class SchemagenIntegerBoundaryTest(unittest.TestCase):
"""Regression tests for issue #19: integer slot parsing near int64 boundaries."""
def setUp(self):
self.repo_root = os.path.dirname(
os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
)
self.include_dir = os.path.join(self.repo_root, "include")
self.lib_src = os.path.join(self.repo_root, "src", "lib.cpp")
self.compiler = shutil.which("c++")
def _compile_harness(self, tmpdir, schema, harness):
schema_path = os.path.join(tmpdir, "schema.json")
with open(schema_path, "w") as fp:
json.dump(schema, fp)
header_path = os.path.join(tmpdir, "gen.h")
result = subprocess.run(
[
sys.executable,
SCRIPT,
schema_path,
"-o",
header_path,
"--namespace",
"test_schema",
],
capture_output=True,
text=True,
check=False,
)
self.assertEqual(result.returncode, 0, msg=result.stderr)
lib_obj = os.path.join(tmpdir, "lib.o")
comp_lib = subprocess.run(
[
self.compiler,
"-std=c++20",
"-I",
self.include_dir,
"-I",
os.path.join(self.repo_root, "third_party", "include"),
"-I",
os.path.join(self.repo_root, "third_party", "valgrind"),
"-c",
self.lib_src,
"-o",
lib_obj,
],
capture_output=True,
text=True,
check=False,
)
self.assertEqual(comp_lib.returncode, 0, msg=comp_lib.stderr)
cpp_path = os.path.join(tmpdir, "test.cpp")
with open(cpp_path, "w") as fp:
fp.write(harness)
exe_path = os.path.join(tmpdir, "test")
comp = subprocess.run(
[
self.compiler,
"-std=c++20",
"-I",
self.include_dir,
"-I",
tmpdir,
lib_obj,
cpp_path,
"-o",
exe_path,
],
capture_output=True,
text=True,
check=False,
)
self.assertEqual(comp.returncode, 0, msg=comp.stderr)
run = subprocess.run([exe_path], capture_output=True, text=True, check=False)
self.assertEqual(run.returncode, 0, msg=run.stdout + run.stderr)
def _build_cases_array(self, name, entries):
lines = [
f" struct {name}Case {{ const char *s; WeaselJsonStatus expected; long long v; }};"
]
lines.append(f" {name}Case {name}_cases[] = {{")
for s, exp, v in entries:
val = f"{v}LL" if isinstance(v, int) else v
lines.append(f' {{ R"({s})", {exp}, {val} }},')
lines.append(" };")
return "\n".join(lines)
def test_integer_boundary_root(self):
if not self.compiler:
self.skipTest("C++ compiler not available")
cases = [
("9223372036854775807.0", "WeaselJson_OK", 9223372036854775807),
("9223372036854775807", "WeaselJson_OK", 9223372036854775807),
("9223372036854775806.0", "WeaselJson_OK", 9223372036854775806),
("9223372036854775808", "WeaselJson_REJECT", 0),
("-9223372036854775808", "WeaselJson_OK", "-9223372036854775807LL - 1"),
("-9223372036854775808.0", "WeaselJson_OK", "-9223372036854775807LL - 1"),
("-9223372036854775809", "WeaselJson_REJECT", 0),
("1e3", "WeaselJson_OK", 1000),
("2.0", "WeaselJson_OK", 2),
("0.001", "WeaselJson_REJECT", 0),
("1e-3", "WeaselJson_REJECT", 0),
("1000e-3", "WeaselJson_OK", 1),
("100.0e-2", "WeaselJson_OK", 1),
("123.0", "WeaselJson_OK", 123),
("9e18", "WeaselJson_OK", 9000000000000000000),
("10e18", "WeaselJson_REJECT", 0),
]
cases_src = self._build_cases_array("root", cases)
harness = textwrap.dedent(
f"""
#include "gen.h"
#include <cstdio>
#include <cstring>
{cases_src}
int main() {{
for (const auto \u0026c : root_cases) {{
test_schema::RootBuilder b;
char buf[512];
std::strncpy(buf, c.s, sizeof(buf) - 1);
buf[sizeof(buf) - 1] = '\\0';
WeaselJsonStatus st = b.feed(buf, std::strlen(buf));
st = b.finish();
if (st != c.expected) {{
std::printf("root case %s expected %d got %d\\n", c.s, c.expected, st);
return 1;
}}
if (st == WeaselJson_OK \u0026\u0026 b.take() != c.v) {{
std::printf("root case %s value mismatch\\n", c.s);
return 2;
}}
}}
return 0;
}}
"""
)
with tempfile.TemporaryDirectory() as tmpdir:
self._compile_harness(tmpdir, {"type": "integer"}, harness)
def test_integer_boundary_object_field(self):
if not self.compiler:
self.skipTest("C++ compiler not available")
cases = [
('{"age":9223372036854775807.0}', "WeaselJson_OK", 9223372036854775807),
('{"age":-9223372036854775809}', "WeaselJson_REJECT", 0),
('{"age":1e3}', "WeaselJson_OK", 1000),
('{"age":2.0}', "WeaselJson_OK", 2),
('{"age":0.001}', "WeaselJson_REJECT", 0),
(
'{"age":-9223372036854775808.0}',
"WeaselJson_OK",
"-9223372036854775807LL - 1",
),
]
cases_src = self._build_cases_array("object", cases)
harness = textwrap.dedent(
f"""
#include "gen.h"
#include <cstdio>
#include <cstring>
{cases_src}
int main() {{
for (const auto \u0026c : object_cases) {{
test_schema::RootBuilder b;
char buf[512];
std::strncpy(buf, c.s, sizeof(buf) - 1);
buf[sizeof(buf) - 1] = '\\0';
WeaselJsonStatus st = b.feed(buf, std::strlen(buf));
st = b.finish();
if (st != c.expected) {{
std::printf("object case %s expected %d got %d\\n", c.s, c.expected, st);
return 3;
}}
if (st == WeaselJson_OK \u0026\u0026 b.take().age != c.v) {{
std::printf("object case %s value mismatch\\n", c.s);
return 4;
}}
}}
return 0;
}}
"""
)
schema = {
"type": "object",
"properties": {"age": {"type": "integer"}},
"required": ["age"],
}
with tempfile.TemporaryDirectory() as tmpdir:
self._compile_harness(tmpdir, schema, harness)
if __name__ == "__main__":
unittest.main()
+181 -77
View File
@@ -60,7 +60,6 @@ class ObjectType:
def __init__(self, name):
self.name = name
self.fields = [] # list[Field]
self.strict = False # additionalProperties: false
class EnumType:
@@ -216,12 +215,26 @@ class Builder:
base = camel(hint)
name = base
i = 1
while name in self._used_names:
while self._name_taken(name):
candidate = f"{base}{i}"
# A numeric suffix can itself land on a reserved generated name
# (e.g. hint "Arr0" -> "Arr01"). Use an underscore separator so
# we never loop through the reserved block.
if self._is_reserved(candidate):
candidate = f"{base}_{i}"
name = candidate
i += 1
name = f"{base}{i}"
self._used_names.add(name)
return name
def _name_taken(self, name):
return name in self._used_names or self._is_reserved(name)
@staticmethod
def _is_reserved(name):
"""Names generated internally that must not collide with user types."""
return name in ("Root", "RootScalar")
def ref_name(self, ref):
if not ref.startswith("#/"):
raise GenError(f"only local $ref supported, got: {ref}")
@@ -237,7 +250,6 @@ class Builder:
if defname not in self.defs:
raise GenError(f"$ref to unknown def: {defname}")
node = self.defs[defname]
# For objects we must register the name before recursing into fields.
return self.build_type(node, defname, defname=defname)
def build_type(self, node, hint, defname=None):
@@ -266,6 +278,10 @@ class Builder:
if "$ref" in node:
return self.build_def(self.ref_name(node["$ref"]))
# Non-object $defs entries must reuse the same type for every $ref.
if defname is not None and defname in self._building:
return self._building[defname]
# nullability via type lists: ["string", "null"]
nullable = False
typ = node.get("type")
@@ -284,18 +300,29 @@ class Builder:
raise GenError("only non-empty string enums are supported")
name = defname and self.unique_name(defname) or self.unique_name(hint)
self.enums[name] = EnumType(name, list(vals))
return (TEnum(name), nullable)
result = (TEnum(name), nullable)
if defname is not None:
self._building[defname] = result
return result
if typ == "object" or (typ is None and "properties" in node):
return (self._build_object(node, hint, defname), nullable)
return self._build_object(node, hint, defname, nullable)
if typ == "array":
if "items" not in node or not isinstance(node["items"], dict):
raise GenError("arrays require a single 'items' schema")
# Register the array before building its items so $ref cycles back
# to this definition resolve to the same TArr instance.
t = TArr(None, False)
result = (t, nullable)
if defname is not None:
self._building[defname] = result
elem, elem_nullable = self._unpack(
self.build_type(node["items"], hint + "Item")
)
return (TArr(elem, elem_nullable), nullable)
t.elem = elem
t.elem_nullable = elem_nullable
return result
scalar = {
"string": "str",
@@ -304,7 +331,10 @@ class Builder:
"boolean": "bool",
}.get(typ)
if scalar:
return (TScalar(scalar), nullable)
result = (TScalar(scalar), nullable)
if defname is not None:
self._building[defname] = result
return result
if typ == "null":
raise GenError("'null'-only types are not supported")
@@ -318,19 +348,21 @@ class Builder:
return result
return (result, False)
def _build_object(self, node, hint, defname):
def _build_object(self, node, hint, defname, nullable=False):
name = self.unique_name(defname or hint)
obj = ObjectType(name)
self.objects[name] = obj
# register for $ref cycles before building fields
tobj = TObj(name)
if defname is not None:
self._building[defname] = TObj(name)
ap = node.get("additionalProperties", True)
self._building[defname] = (tobj, nullable)
ap = node.get("additionalProperties", False)
if ap is True:
raise GenError("additionalProperties: true is not supported")
if isinstance(ap, dict):
raise GenError(
"additionalProperties with a schema (typed map) is not " "supported yet"
)
obj.strict = ap is False
required = set(node.get("required", []))
props = node.get("properties", {})
seen_cpp = set()
@@ -344,7 +376,7 @@ class Builder:
cpp = f"{base}{i}"
seen_cpp.add(cpp)
obj.fields.append(Field(key, cpp, ty, key in required, nullable))
return TObj(name)
return tobj
# ---------------------------------------------------------------------------
@@ -435,6 +467,7 @@ class Emitter:
self.kind_order = [] # all Kind enumerators in declaration order
self.root_ty = None
self.root_nullable = False
self._arr_counter = 0
# -- type strings -------------------------------------------------------
def base_cpp(self, ty):
@@ -465,11 +498,19 @@ class Emitter:
def arr_kind(self, tarr):
sig = self.base_cpp(tarr)
if sig not in self.arr_kinds:
name = f"Arr{len(self.arr_kinds)}"
name = self._fresh_arr_kind_name()
self.arr_kinds[sig] = name
self.arr_types.append((name, tarr))
self.b._used_names.add(name)
return self.arr_kinds[sig]
def _fresh_arr_kind_name(self):
while True:
name = f"Arr{self._arr_counter}"
self._arr_counter += 1
if not self.b._name_taken(name):
return name
def cat(self, ty):
if isinstance(ty, TScalar):
return {"str": "Str", "int": "Int", "dbl": "Dbl", "bool": "Bool"}[ty.kind]
@@ -493,6 +534,18 @@ class Emitter:
self.root_ty, self.root_nullable = Builder._unpack(
self.b.build_type(self.b.root_schema, "Root")
)
# A nullable root object would otherwise produce
# using Root = std::optional<Root>;
# which conflicts with the struct named Root. Rename the inner struct.
if isinstance(self.root_ty, TObj) and self.root_nullable:
old_name = self.root_ty.name
new_name = self.b.unique_name("RootInner")
obj = self.b.objects.pop(old_name)
obj.name = new_name
self.b.objects[new_name] = obj
self.root_ty = TObj(new_name)
break_cycles(self.b.objects)
# register all array kinds (walk every field + root)
@@ -582,20 +635,17 @@ namespace {ns} {{"""
def _kind_enum(self):
kinds = list(self.b.objects.keys())
kinds += [n for n, _ in self.arr_types]
root_is_container = (
isinstance(self.root_ty, (TObj, TArr)) and not self.root_nullable
)
root_is_container = isinstance(self.root_ty, (TObj, TArr))
if not root_is_container:
kinds.append("RootScalar")
kinds.append("Skip")
self.kind_order = kinds
return " enum class Kind : uint8_t { " + ", ".join(kinds) + " };"
def _root_info(self):
"""Return (root_cat, root_container_kind_or_None)."""
if isinstance(self.root_ty, TObj) and not self.root_nullable:
if isinstance(self.root_ty, TObj):
return ("Obj", f"Kind::{self.root_ty.name}")
if isinstance(self.root_ty, TArr) and not self.root_nullable:
if isinstance(self.root_ty, TArr):
return ("Arr", f"Kind::{self.arr_kind(self.root_ty)}")
return (self.cat(self.root_ty), None)
@@ -665,6 +715,11 @@ namespace {ns} {{"""
lines.append(" }")
root_cat, root_kind = self._root_info()
if root_kind is None:
if self.root_nullable:
lines.append(
" case Kind::RootScalar: { if (!result_) result_.emplace(); return &*result_; }"
)
else:
lines.append(" case Kind::RootScalar: return &result_;")
lines.append(" default: return nullptr;")
lines.append(" }")
@@ -771,17 +826,6 @@ namespace {ns} {{"""
lines.append(" }")
return "\n".join(lines)
def _is_strict(self):
strict = [n for n, o in self.b.objects.items() if o.strict]
if not strict:
return " bool isStrict(Kind) const { return false; }"
cases = " ".join(f"case Kind::{n}:" for n in strict)
return (
" bool isStrict(Kind k) const {\n"
f" switch (k) {{ {cases} return true; default: return false; }}\n"
" }"
)
def _enum_name_arrays(self):
out = []
for e in self.b.enums.values():
@@ -807,7 +851,12 @@ namespace {ns} {{"""
lines = [" if (error_) return;"]
if root_kind is not None and root_cat == event_cat:
lines.append(" if (stack_.empty()) {")
lines.append(f" stack_.push_back(Frame{{{root_kind}, &result_}});")
if self.root_nullable:
lines.append(" result_.emplace();")
dest = "&*result_"
else:
dest = "&result_"
lines.append(f" stack_.push_back(Frame{{{root_kind}, {dest}}});")
if event_cat == "Obj":
lines.append(" {")
lines.append(f" int n = fieldCount({root_kind});")
@@ -818,13 +867,7 @@ namespace {ns} {{"""
else:
lines.append(" if (stack_.empty()) { reject(); return; }")
lines.append(" Frame &f = stack_.back();")
lines.append(
" if (f.kind == Kind::Skip) { stack_.push_back(Frame{Kind::Skip, nullptr}); return; }"
)
lines.append(" SlotInfo si = slotInfoG(f);")
lines.append(
" if (si.cat == Cat::Skip) { stack_.push_back(Frame{Kind::Skip, nullptr}); return; }"
)
lines.append(f" if (si.cat != Cat::{event_cat}) {{ reject(); return; }}")
lines.append(" void *p = engage(f, true);")
lines.append(" stack_.push_back(Frame{si.child, p});")
@@ -839,6 +882,9 @@ namespace {ns} {{"""
root_cat, root_kind = self._root_info()
begin_obj = self._begin_container("Obj", root_kind)
begin_arr = self._begin_container("Arr", root_kind)
null_at_root = (
"done_ = true; return;" if self.root_nullable else "reject(); return;"
)
return f"""
class RootBuilder {{
@@ -872,7 +918,7 @@ public:
private:
{kind_enum}
enum class Cat {{ Reject, Str, Int, Dbl, Bool, Enum, Obj, Arr, Skip }};
enum class Cat {{ Reject, Str, Int, Dbl, Bool, Enum, Obj, Arr }};
struct SlotInfo {{
Cat cat;
Kind child{{}};
@@ -883,11 +929,10 @@ private:
struct Frame {{
Kind kind;
void *dest;
int field = -1; // object: selected field (-1 want key, -2 skip)
int field = -1; // object: selected field (-1 means waiting for key)
std::vector<uint64_t> seen; // populated field bitset (object frames)
}};
static constexpr int kWantKey = -1;
static constexpr int kSkip = -2;
{self._enum_name_arrays()}
@@ -902,10 +947,93 @@ private:
void reject() {{ error_ = true; }}
// Wrap the generated slotInfo() with the generic key/skip states.
// Parse a JSON number text as int64_t. Accepts optional decimal point and
// exponent only when the mathematical value is an integer that fits in a
// signed 64-bit range.
static bool parseJsonInt64(const char *b, const char *e, int64_t &out) {{
const char *p = b;
bool neg = false;
if (p < e) {{
if (*p == '-') {{ neg = true; ++p; }}
else if (*p == '+') return false;
}}
const char *intStart = p;
while (p < e && *p >= '0' && *p <= '9') ++p;
const char *intEnd = p;
int fracDigits = 0;
const char *fracStart = p;
if (p < e && *p == '.') {{
++p;
fracStart = p;
while (p < e && *p >= '0' && *p <= '9') {{ ++p; ++fracDigits; }}
if (fracStart == p) return false;
}}
int64_t exp = 0;
bool expNeg = false;
if (p < e && (*p == 'e' || *p == 'E')) {{
++p;
if (p < e && (*p == '-' || *p == '+')) {{ expNeg = (*p == '-'); ++p; }}
if (p == e || *p < '0' || *p > '9') return false;
while (p < e && *p >= '0' && *p <= '9') {{
int digit = *p - '0';
if (exp <= (INT64_MAX - digit) / 10) exp = exp * 10 + digit;
else exp = INT64_MAX;
++p;
}}
if (expNeg) exp = -exp;
}}
if (p != e) return false;
if (intStart == intEnd) return false;
std::string digits;
digits.reserve((intEnd - intStart) + fracDigits);
for (const char *q = intStart; q < intEnd; ++q) digits.push_back(*q);
for (int i = 0; i < fracDigits; ++i) digits.push_back(fracStart[i]);
int64_t trim = 0;
while (!digits.empty() && digits.back() == '0') {{
digits.pop_back();
++trim;
}}
int64_t finalExp = exp - fracDigits + trim;
if (finalExp < 0) return false;
while (!digits.empty() && digits.front() == '0') digits.erase(digits.begin());
if (digits.empty()) {{ out = 0; return true; }}
constexpr uint64_t kMaxNeg = 9223372036854775808ULL;
constexpr uint64_t kMaxPos = 9223372036854775807ULL;
const uint64_t limit = neg ? kMaxNeg : kMaxPos;
if (finalExp > 19) return false;
int64_t maxSig = 19 - finalExp;
uint64_t mag = 0;
int64_t sigDigits = 0;
for (char ch : digits) {{
uint64_t d = static_cast<uint64_t>(ch - '0');
if (sigDigits >= maxSig) return false;
if (mag > (limit - d) / 10) return false;
mag = mag * 10 + d;
++sigDigits;
}}
for (int64_t i = 0; i < finalExp; ++i) {{
if (mag > limit / 10) return false;
mag *= 10;
}}
if (mag > limit) return false;
if (neg) {{
if (mag == kMaxNeg) {{
out = INT64_MIN;
}} else {{
out = -static_cast<int64_t>(mag);
}}
}} else {{
out = static_cast<int64_t>(mag);
}}
return true;
}}
// Wrap the generated slotInfo() with the generic key state.
SlotInfo slotInfoG(const Frame &f) {{
if (isObjectKind(f.kind)) {{
if (f.field == kSkip) return SlotInfo{{Cat::Skip}};
if (f.field < 0) return SlotInfo{{Cat::Reject}};
}}
return slotInfo(f);
@@ -930,11 +1058,6 @@ private:
void cbEndObject() {{
if (error_) return;
Frame &f = stack_.back();
if (f.kind == Kind::Skip) {{
stack_.pop_back();
if (stack_.empty() || stack_.back().kind != Kind::Skip) valueComplete();
return;
}}
if (isObjectKind(f.kind)) {{
const auto &req = requiredMask(f.kind);
bool missing = false;
@@ -952,11 +1075,6 @@ private:
void cbEndArray() {{
if (error_) return;
Frame f = stack_.back();
if (f.kind == Kind::Skip) {{
stack_.pop_back();
if (stack_.empty() || stack_.back().kind != Kind::Skip) valueComplete();
return;
}}
stack_.pop_back();
valueComplete();
}}
@@ -968,12 +1086,10 @@ private:
scratch_.append(buf, len);
if (!done) return;
int idx = matchKey(f.kind, scratch_);
scratch_.clear();
if (idx < 0) {{
if (isStrict(f.kind)) {{ reject(); return; }}
f.field = kSkip;
return;
reject(); return; // unknown key
}}
scratch_.clear();
if (f.seen[idx >> 6] & (1ull << (idx & 63))) {{
reject(); return; // duplicate key
}}
@@ -983,9 +1099,7 @@ private:
void cbStringData(const char *buf, int len, int done) {{
if (error_) return;
Frame &f = stack_.back();
if (f.kind == Kind::Skip) return;
SlotInfo si = slotInfoG(f);
if (si.cat == Cat::Skip) {{ if (done) valueComplete(); return; }}
if (si.cat == Cat::Str) {{
auto *s = (std::string *)engage(f, !started_);
s->append(buf, len);
@@ -1011,9 +1125,7 @@ private:
void cbNumberData(const char *buf, int len, int done) {{
if (error_) return;
Frame &f = stack_.back();
if (f.kind == Kind::Skip) return;
SlotInfo si = slotInfoG(f);
if (si.cat == Cat::Skip) {{ if (done) valueComplete(); return; }}
if (si.cat != Cat::Int && si.cat != Cat::Dbl) {{ reject(); return; }}
scratch_.append(buf, len);
started_ = true;
@@ -1028,16 +1140,11 @@ private:
*(int64_t *)p = v; // plain integer literal: parsed exactly
}} else {{
// JSON Schema "integer" accepts any number with no fractional part,
// including exponent/decimal forms like 1e3 or 2.0. Parse those as a
// double and require an integral value within int64 range.
double d = 0;
auto rd = std::from_chars(b, e, d);
if (rd.ec != std::errc() || rd.ptr != e || d != std::trunc(d) ||
d < -9223372036854775808.0 || d >= 9223372036854775808.0) {{
reject();
return;
}}
*(int64_t *)p = (int64_t)d;
// including exponent/decimal forms like 1e3 or 2.0. Parse those
// exactly as int64 when the value is integral and in range.
int64_t v2 = 0;
if (!parseJsonInt64(b, e, v2)) {{ reject(); return; }}
*(int64_t *)p = v2;
}}
}} else {{
double v = 0;
@@ -1051,9 +1158,7 @@ private:
void cbBool(bool value) {{
if (error_) return;
Frame &f = stack_.back();
if (f.kind == Kind::Skip) return;
SlotInfo si = slotInfoG(f);
if (si.cat == Cat::Skip) {{ valueComplete(); return; }}
if (si.cat != Cat::Bool) {{ reject(); return; }}
*(bool *)engage(f, true) = value;
valueComplete();
@@ -1065,10 +1170,11 @@ private:
void cbNull() {{
if (error_) return;
if (stack_.empty()) {{
{null_at_root}
}}
Frame &f = stack_.back();
if (f.kind == Kind::Skip) return;
SlotInfo si = slotInfoG(f);
if (si.cat == Cat::Skip) {{ valueComplete(); return; }}
if (!si.nullable) {{ reject(); return; }}
if (isArrayKind(f.kind)) appendNull(f); // keep null array elements
valueComplete(); // leave optional empty / pointer null
@@ -1106,8 +1212,6 @@ private:
{self._reqmask()}
{self._is_object_kind()}
{self._is_strict()}
}};
"""