X-CGP-ClamAV-Result: CLEAN X-VirusScanner: Niversoft's CGPClamav Helper v1.22.2a (ClamAV engine v0.102.2) X-Junk-Score: 0 [] X-KAS-Score: 0 [] From: "OCsite" Received: from smtp-beta-1.zoner.com ([217.198.120.66] verified) by post.selbstdenker.com (CommuniGate Pro SMTP 6.3.3) with ESMTPS id 25648047 for webobjects-dev@wocommunity.org; Tue, 30 Mar 2021 15:43:03 +0200 Received-SPF: none receiver=post.selbstdenker.com; client-ip=217.198.120.66; envelope-from=ocs@ocs.cz Received: from smtp.zoner.com (smtp.zoner.com [217.198.120.6]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp-beta-1.zoner.com (Postfix) with ESMTPS id B30AC180008F for ; Tue, 30 Mar 2021 15:42:42 +0200 (CEST) Received: from [172.19.31.202] (unknown [46.167.220.254]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) (Authenticated sender: ocs@ocs.cz) by smtp.zoner.com (Postfix) with ESMTPSA id 68EA73000077 for ; Tue, 30 Mar 2021 15:42:42 +0200 (CEST) Content-Type: multipart/alternative; boundary="Apple-Mail=_05C0E247-E8FA-48C9-BA1D-D48C459EDE0F" Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.120.23.2.4\)) Subject: Re: [WO-DEV] ERXObjectStoreCoordinatorSynchronizer woes Date: Tue, 30 Mar 2021 15:42:41 +0200 References: To: WebObjects & WOnder Development In-Reply-To: Message-Id: X-Mailer: Apple Mail (2.3608.120.23.2.4) --Apple-Mail=_05C0E247-E8FA-48C9-BA1D-D48C459EDE0F Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Ramsey, thanks! > if you are using ERXObjectStoreCoordinatorPool We used to, but currently we do not. We are just setting = ERXObjectStoreCoordinatorPool.maxCoordinators to 1 to run the = CoordinatorSynchronizer (to sync our occasionally manually created = object stores), and as I wrote lately, I am considering to switch even = that off: > I guess we can switch off the Coordinator completely and coordinate = manually: we use background threads with their own EO stacks pretty = rarely and for a very small set of well-defined operations (the = aforementioned import, not-yet mentioned pre-computing of some cached = data, and that's all), and we never use separate EO stacks for normal = workers; thus a manual coordination would not be too difficult. As for memory leaks, my current approach is essentially this: =3D=3D=3D importThread { ERXObjectStoreCoordinator objectStore=3Dnew = ERXObjectStoreCoordinator(YES) objectStore.lock() // without this, CoordinatorSynchronizer sometimes = caused "FATAL Unlocking thread is not locking thread" ERXEC ec=3DERXEC.newEditingContext(objectStore) ... import and save into ec ... ec=3Dnull // probably superfluous, can't harm objectStore.unlock() Thread.sleep(60000) // without this, CoordinatorSynchronizer sometimes = caused NPEs objectStore.dispose() // to free the DB socket, compare also thread = "Looks like an OSC must be disposed manually?!?" of 29/6/20 } =3D=3D=3D So far, this seems to work more or less properly. What's weird, the OSCs = do not get always garbage-collected; when testing and thus doing many = imports I have found about third to half of them remains disposed, = connection to DB is closed all right, but not finalised and still = occupying memory. Since normally, just a few imports happen before the application is = restarted, it is not a big problem; but if you can see the culprit, I'll = gladly fix it :) Thanks again and all the best, OC > On 29 Mar 2021, at 2:22, Ramsey Gurley = wrote: >=20 > Just looking at your issue here, >=20 >> On Mar 21, 2021, at 1:01 PM, OCsite > wrote: >>=20 >> 04:38:38.600 ERROR java.lang.NullPointerException = //log:er.extensions.eof.ERXObjectStoreCoordinatorSynchronizer = [ERXOSCProcessChanges] >> NullPointerException >> at = com.webobjects.eoaccess.EOModelGroup.modelGroupForObjectStoreCoordinator(E= OModelGroup.java:795) >> at = er.extensions.eof.ERXEOAccessUtilities.databaseContextForEntityNamed(ERXEO= AccessUtilities.java:1086) >> at = er.extensions.eof.ERXObjectStoreCoordinatorSynchronizer$ProcessChangesQueu= e._process(ERXObjectStoreCoordinatorSynchronizer.java:509) >> at = er.extensions.eof.ERXObjectStoreCoordinatorSynchronizer$ProcessChangesQueu= e.process(ERXObjectStoreCoordinatorSynchronizer.java:540) >=20 > This last line seems to be where your problem is coming from. If you = look in ERXObjectStoreCoordinatorSynchronizer, you see that = _synchronizer.coordinators() enumeration must be returning null, which = it then passes into _process. >=20 > protected void process(EOObjectStoreCoordinator sender, = SnapshotProcessor processor, NSDictionary changesByEntity, String = userInfoKey) { > NSMutableDictionary dbcs =3D new NSMutableDictionary(); > for (Enumeration oscs =3D _synchronizer.coordinators(); = oscs.hasMoreElements();) { > EOObjectStoreCoordinator osc =3D (EOObjectStoreCoordinator) = oscs.nextElement(); > if (osc !=3D sender) { > _process(sender, osc, dbcs, processor, changesByEntity, userInfoKey); > } > } > } >=20 > It's using unsynchronized access to a private mutable array, which is = probably how it is null in the first place. Maybe a race condition. = Seems like if that was changed to >=20 > if(osc !=3D null && osc !=3D sender) >=20 > then this specific problem might go away. >=20 > But just as a side note, if you are using = ERXObjectStoreCoordinatorPool, I discovered it leaks memory when I was = working on persistent session storage about 9 years ago. The leak did = not appear to be fixable without rewriting EOEditingContext. >=20 > BTW, if anyone is looking for a full remote WO job and you love the = wild world of accountants, we still have an opening for a senior = developer. Get in touch if you want to inherit a WO app backend. The = frontend is done in ReactJS and maintained by someone else, so all you = really need to do is take care of the backend through direct action = endpoints. You probably also get to help me migrate another WO app to a = React frontend in the future. >=20 > = https://www.indeed.com/jobs?q=3Dprosites&from=3Dgooglesl&vjk=3Dc43301acc8d= db2a0 = >=20 > Thanks, >=20 > Ramsey > Confidentiality Notice: This email, including all attachments and = replies thereto, are covered by the Electronic Communications Privacy = Act, 18 U.S.C. Sections 2510-2521 and are legally privileged. This = information is confidential, and intended only for the use of the = individuals or entities named above. If you are not the intended = recipient, you are hereby notified that any disclosure, copying, = distribution or the taking of any action in reliance on the contents of = this transmitted information is strictly prohibited. Please notify us if = you have received this transmission in error. Thank you. --Apple-Mail=_05C0E247-E8FA-48C9-BA1D-D48C459EDE0F Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii Ramsey,

thanks!

if you are using = ERXObjectStoreCoordinatorPool

We used to, but currently we do not. We are just = setting ERXObjectStoreCoordinatorPool.maxCoordinators to 1 to run = the CoordinatorSynchronizer (to sync our = occasionally manually created object stores), and as I wrote lately, I = am considering to switch even that off:

I = guess we can switch off the Coordinator completely and coordinate = manually: we use background threads with their own EO stacks pretty = rarely and for a very small set of well-defined operations (the = aforementioned import, not-yet mentioned pre-computing of some cached = data, and that's all), and we never use separate EO stacks for normal = workers; thus a manual coordination would not be too = difficult.

As for = memory leaks, my current approach is essentially this:

=3D=3D=3D
importThread {
  ERXObjectStoreCoordinator = objectStore=3Dnew ERXObjectStoreCoordinator(YES)
<= div class=3D"">  objectStore.lock() // without this, = CoordinatorSynchronizer sometimes caused "FATAL Unlocking thread is not = locking thread"
  ERXEC = ec=3DERXEC.newEditingContext(objectStore)
  ...  import and save into ec = ...
  ec=3Dnull = // probably superfluous, can't harm
  objectStore.unlock()
  Thread.sleep(60000) // without this, = CoordinatorSynchronizer sometimes caused NPEs
  objectStore.dispose() // to free the DB socket, = compare also thread "Looks like an OSC must be disposed manually?!?" of = 29/6/20
}
=3D=3D=3D

So far, this seems to = work more or less properly. What's weird, the OSCs do not = get always garbage-collected; when testing and thus doing many = imports I have found about third to half of them remains disposed, = connection to DB is closed all right, but not finalised and still = occupying memory.

Since normally, just a few imports happen before the = application is restarted, it is not a big problem; but if you can see = the culprit, I'll gladly fix it :)

Thanks again and all the best,
OC

On 29 Mar 2021, at 2:22, Ramsey Gurley <webobjects-dev@wocommunity.org> wrote:

Just looking at your issue here,

On Mar 21, 2021, at 1:01 PM, OCsite <webobjects-dev@wocommunity.org> wrote:

04:38:38.600 ERROR java.lang.NullPointerException =       = //log:er.extensions.eof.ERXObjectStoreCoordinatorSynchronizer = [ERXOSCProcessChanges]
NullPointerException
  at = com.webobjects.eoaccess.EOModelGroup.modelGroupForObjectStoreCoordinator(E= OModelGroup.java:795)
  at = er.extensions.eof.ERXEOAccessUtilities.databaseContextForEntityNamed(ERXEO= AccessUtilities.java:1086)
  at = er.extensions.eof.ERXObjectStoreCoordinatorSynchronizer$ProcessChangesQueu= e._process(ERXObjectStoreCoordinatorSynchronizer.java:509)
  at = er.extensions.eof.ERXObjectStoreCoordinatorSynchronizer$ProcessChangesQueu= e.process(ERXObjectStoreCoordinatorSynchronizer.java:540)

This last line seems to be where your problem is coming = from. If you look in ERXObjectStoreCoordinatorSynchronizer, you see that = _synchronizer.coordinators() enumeration must be returning null, which = it then passes into _process.

protected void = process(EOObjectStoreCoordinator sender, = SnapshotProcessor processor, NSDictionary changesByEntity, String userInfoKey) {
NSMutableDictionary dbcs =3D new NSMutableDictionary();
for (Enumeration oscs =3D _synchronizer.coordinators(); oscs.hasMoreElements();) {
EOObjectStoreCoordinator osc =3D = (EOObjectStoreCoordinator) oscs.nextElement();
if (osc !=3D sender) {
_process(sender, osc, dbcs, processor, = changesByEntity, userInfoKey);
}
}
}

It's using unsynchronized access to a private mutable array, = which is probably how it is null in the first place. Maybe a race = condition. Seems like if that was changed to

if(osc !=3D null && osc !=3D sender)

then this specific problem might go away.

But just as a side note, if you are using = ERXObjectStoreCoordinatorPool, I discovered it leaks memory when I was = working on persistent session storage about 9 years ago. The leak did not appear to be fixable without rewriting EOEditingContext.

BTW, if anyone is looking for a full remote WO job and you = love the wild world of accountants, we still have an opening for a = senior developer. Get in touch if you want to inherit a WO app backend. The frontend is done in ReactJS and maintained by = someone else, so all you really need to do is take care of the backend = through direct action endpoints. You probably also get to help me = migrate another WO app to a React frontend in the future.


Thanks,

Ramsey
Confidentiality Notice: This email, = including all attachments and replies thereto, are covered by the = Electronic Communications Privacy Act, 18 U.S.C. Sections 2510-2521 and = are legally privileged. This information is confidential, and intended = only for the use of the individuals or entities named above. If you are not = the intended recipient, you are hereby notified that any disclosure, = copying, distribution or the taking of any action in reliance on the = contents of this transmitted information is strictly prohibited. Please notify us if you have received this transmission in = error. Thank you.

= --Apple-Mail=_05C0E247-E8FA-48C9-BA1D-D48C459EDE0F--