Date: Thu, 28 Mar 2024 22:54:27 +0000 (UTC) Message-ID: <518595898.7141.1711666467342@ip-10-10-7-29.ec2.internal> Subject: Exported From Confluence MIME-Version: 1.0 Content-Type: multipart/related; boundary="----=_Part_7140_1361875158.1711666467340" ------=_Part_7140_1361875158.1711666467340 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Content-Location: file:///C:/exported.html
json-lib is a dead project, is very slow, is a memory hog, and h= as memory leaks. We will migrate to jackson.
In 4.10.1+ and 5.7.0+ Jackson marshaling is used by default but will fai= lover to json-lib if invalid attributes in JSON are used.
Example: https://gith= ub.com/Internet2/grouper/commit/ef4961ebf298b06f366217f73468138089dfe8e9
Old way with json-lib
FROM
JSONObj= ect jsonObject =3D (JSONObject) JSONSerializer.toJSON( body );
That has an object model that is obviously different than jackson, but i= t is similar. Convert to map instead, or use the jackson parsing obje= ct model. In this case we will try the object model
TO
JsonNod= e jsonObject =3D GrouperUtil.jsonJacksonNode(body);
Change calls, e.g.
FROM
d= ebugMap.put("totalSize", jsonObject.getString("totalSize"));
TO
d= ebugMap.put("totalSize", GrouperUtil.jsonJacksonGetString(jsonObject, "tota= lSize"));
FROM
= String userId =3D jsonObjectUser.getString("userId");
TO (be careful about NPE if getting ints!)
= String userId =3D GrouperUtil.jsonJacksonGetString(jsonObjectUser, "userId= ");
FROM
= JSONObject userWithGroupsJson =3D new JSONObject(); JSONArray groupsJson =3D new JSONArray();
TO
= ObjectNode userWithGroupsJson =3D GrouperUtil.jsonJacksonNode(); ArrayNode groupsJson =3D GrouperUtil.jsonJacksonArra= yNode();
FROM
jsonObj= ect.containsKey("id")
TO
jsonObj= ect.has("id")
FROM
jsonObj= ect.getArrayNode("groups")
TO
(ArrayN= ode)jsonObject.get("groups")
FROM
j= sonObject.put("name", name);
TO
G= rouperUtil.jsonJacksonAssignString(jsonObject, "name", name);
FROM
jsonObj= ect.toString();
TO
Grouper= Util.jsonJacksonToString(jsonObject);
Results
json-li= b took: 3081ms, found 2000 records jackson took: 14ms, found 2000 records jackson map took: 51ms, found 2000 records
First problem is Penn's daemon keeps crashing when running the remedy di= gital marketplace job that uses json-lib. Not sure what changed, mayb= e there are more users to download now?
For the remedy digital marketplace connector, it downloads all users in = batches of 2000. This is a 2Meg json string. It takes 1.5 seconds to do the= WS call from the app. Then it takes 2 minutes to convert that into a JSON = object.
2020-11= -12 11:51:55,332: [main] DEBUG GrouperDigitalMarketplaceLog.marketplaceLog(= 43) - - method: retrieveDigitalMarketplaceUsersHelper, authnMillis: 229ms, = getMillis: 1511ms, totalSize: 128379, first: 43253.0, last: zzimm, elapsedM= illis: 129512
Heres a memory leak report
https://sourceforge.ne=
t/p/json-lib/bugs/search/?q=3Dmemory
New provisioning connectors shouldnt use it. Eventually we need to move awa=
y from it for web services too.
If I set the paging size down to 200 instead of 2000 it seems like it mi= ght work, but we still need to swap this out. 340ms to make rest call and 7= 0ms to parse json...
2020-11= -12 12:24:34,113: [main] DEBUG GrouperDigitalMarketplaceLog.marketplaceLog(= 43) - - method: retrieveDigitalMarketplaceUsersHelper, getMillis: 340ms, to= talSize: 128379, first: aahuja77, last: zarany, elapsedMillis: 409
well, its not consistent. sometimes it takes longer. this says 300ms to = call WS and 2.5 seconds to parse json
2020-11= -12 12:27:24,069: [main] DEBUG GrouperDigitalMarketplaceLog.marketplaceLog(= 43) - - method: retrieveDigitalMarketplaceUsersHelper, getMillis: 298ms, to= talSize: 128379, first: 43256.0, last: zyuqing, elapsedMillis: 2950
Its going up by a factor of 10 as it runs
Here is 300ms for call, and 13 seconds to parse
2020-11= -12 12:37:05,796: [main] DEBUG GrouperDigitalMarketplaceLog.marketplaceLog(= 43) - - method: retrieveDigitalMarketplaceUsersHelper, getMillis: 369ms, to= talSize: 128379, first: adambis, last: zampou, elapsedMillis: 13262
Heres a datapoint parsing the 2meg json
json-li= b took: 3471ms, found 2000 records jackson took: 180ms, found 2000 records
com.googlecode.json-simple
https://search.maven=
.org/artifact/org.json/json/20140107/jar ( or newer )
And I know I use jackson around here somewhere too=E2=80=A6.
FWIW: I just added a new CLC that is using json-lib to my custom stack=E2=
=80=A6
So=E2=80=A6 Please make it clear when that dependency evaporates. ( I will =
likely need to convert away from json-lib in a project or two around here=
=E2=80=A6.)
Note I have not done any performance/feature comparisons against the libs=
=E2=80=A6 YMMV. (edited)
OK, thanks for the pointer. That one looks equivalent to jackson, and we= already have jackson, so Im inclined to keep using that one. I dont know w= hen json-lib would be taken out of grouper, you have some time https://www.overops.com/blog/th= e-ultimate-json-library-json-simple-vs-gson-vs-jackson-vs-json/