X-CGP-ClamAV-Result: CLEAN X-VirusScanner: Niversoft's CGPClamav Helper v1.22.2a (ClamAV engine v0.102.2) From: "Maik Musall" Received: from [81.27.162.131] (account maik@selbstdenker.ag HELO [81.27.162.226]) by selbstdenker.ag (CommuniGate Pro SMTP 6.3.3) with ESMTPSA id 25384135 for webobjects-dev@wocommunity.org; Tue, 09 Feb 2021 17:03:13 +0100 Content-Type: multipart/signed; boundary="Apple-Mail=_7CA46C54-5EE9-4849-8854-4011F8F05925"; protocol="application/pkcs7-signature"; micalg=sha-256 Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.60.0.2.21\)) Subject: Re: [WO-DEV] Urgent Out of Memory help needed Date: Tue, 9 Feb 2021 17:03:12 +0100 References: To: WebObjects & WOnder Development In-Reply-To: Message-Id: <42240929-DB74-4D28-9554-D2EB9240E8E5@selbstdenker.ag> X-Mailer: Apple Mail (2.3654.60.0.2.21) --Apple-Mail=_7CA46C54-5EE9-4849-8854-4011F8F05925 Content-Type: multipart/alternative; boundary="Apple-Mail=_A5BF7D4C-0015-420A-914C-03279A09E749" --Apple-Mail=_A5BF7D4C-0015-420A-914C-03279A09E749 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Hi Markus, you upped the intance *to* 2GB? That=E2=80=99s still Raspberry Pi = territory=E2=80=A6 I suspect some Javascript that doesn=E2=80=99t hit the existing session = and creates new ones on every request. I=E2=80=99d add logging stack = traces for new sessions to see where they come from. Maik public static void logStackTrace( String message, int maxSteps ) = { StringBuilder sb =3D new StringBuilder(); if( message !=3D null ) sb.append( message ); sb.append( " from\n" ); int m =3D 0; final int log=46rom =3D 1; final int maxStackDepth =3D log=46rom + maxSteps; for( StackTraceElement stackElem : new = Throwable().getStackTrace() ) { if( m >=3D log=46rom ) sb.append( = stackElem.toString() ).append( "\n" ); if( ++m >=3D maxStackDepth ) break; } log.info( sb.toString() ); } > Am 08.02.2021 um 09:42 schrieb Markus Ruggiero (rucotec) = : >=20 > We are running the same application code in a handful of instances. = Each instance is dedicated to one customer. There are only some very = small differences between the instances like db credentials and some = webserver resources (customer specific logo etc). One customer is = driving their dedicated instance regularly into java memory exhaustion, = all others have no problem. This particular customer is using one screen = more than other customers do. This screen has a couple Ajax components, = but as far as I can tell nothing too fancy. In WOMonitor we can see that = the Running Session count at the bottom right of the details display = goes up like crazy. Within hours the count can reach 1500+. I started to = trace all Session creation (Thread.dumpStack() in Session constructor) = but as expected there was exactly one session being created at login, = then none more.=20 >=20 > What the heck is going on here? What does WOMonitor display here? = Obviously must be something other than just session creation. Could this = be an Ajax issue? Where would you start? We upped the JVM memory to 2G = to survive the day and have more instances for that particular customer = running. >=20 > Btw: how do you go into WOStatistics? Setting a statistics password in = WOMonitor did not help, the direct action always comes back with a = browser error (Safari can=E2=80=99t open the page bla bla bla). I seem = the miss something here. >=20 > Thanks for any help > ---markus--- >=20 > Markus Ruggiero >=20 > rucotec GmbH web https://rucotec.ch = > Steinenvorstadt 79 email markus.ruggiero@rucotec.ch = > 4051 Basel / Switzerland mobile +41 79 508 4701 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 --Apple-Mail=_A5BF7D4C-0015-420A-914C-03279A09E749 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 Hi = Markus,

you upped = the intance *to* 2GB? That=E2=80=99s still Raspberry Pi = territory=E2=80=A6

I suspect some Javascript that doesn=E2=80=99t hit the = existing session and creates new ones on every request. I=E2=80=99d add = logging stack traces for new sessions to see where they come = from.

Maik

public static void logStackTrace( String message, int maxSteps ) {
= StringBuilder sb = =3D new = StringBuilder();
if( message !=3D null ) sb.append( message );
= sb.append( " from\n" );
= int m =3D 0;
= final int logFrom =3D 1;
= final int = maxStackDepth =3D = logFrom + = maxSteps;
for( StackTraceElement stackElem : new Throwable().getStackTrace() ) {
= if( = m >=3D logFrom ) sb.append( stackElem.toString() ).append( "\n" );
= if( ++m >=3D maxStackDepth ) break;
= }
log.info( sb.toString() );
= }


Am = 08.02.2021 um 09:42 schrieb Markus Ruggiero (rucotec) <webobjects-dev@wocommunity.org>:

We are running the = same application code in a handful of instances. Each instance is = dedicated to one customer. There are only some very small differences = between the instances like db credentials and some webserver resources = (customer specific logo etc). One customer is driving their dedicated = instance regularly into java memory exhaustion, all others have no = problem. This particular customer is using one screen more than other = customers do. This screen has a couple Ajax components, but as far as I = can tell nothing too fancy. In WOMonitor we can see that the Running = Session count at the bottom right of the details display goes up like = crazy. Within hours the count can reach 1500+. I started to trace all = Session creation (Thread.dumpStack() in Session constructor) but as = expected there was exactly one session being created at login, then none = more. 

What the = heck is going on here? What does WOMonitor display here? Obviously must = be something other than just session creation. Could this be an Ajax = issue? Where would you start? We upped the JVM memory to 2G to survive = the day and have more instances for that particular customer = running.

Btw: = how do you go into WOStatistics? Setting a statistics password in = WOMonitor did not help, the direct action always comes back with a = browser error (Safari can=E2=80=99t open the page bla bla bla). I seem = the miss something here.

Thanks for any help
---markus---

Markus = Ruggiero

rucotec GmbH              =           web https://rucotec.ch
Steinenvorstadt 79          =       email markus.ruggiero@rucotec.ch
4051 Basel / = Switzerland         mobile +41 79 508 4701









= --Apple-Mail=_A5BF7D4C-0015-420A-914C-03279A09E749-- --Apple-Mail=_7CA46C54-5EE9-4849-8854-4011F8F05925 Content-Disposition: attachment; filename=smime.p7s Content-Type: application/pkcs7-signature; name=smime.p7s Content-Transfer-Encoding: base64 MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgEFADCABgkqhkiG9w0BBwEAAKCCDgow gga4MIIEoKADAgECAg8ZF5XcInQbEh3bVExcy9wwDQYJKoZIhvcNAQELBQAwRTELMAkGA1UEBhMC Q0gxFTATBgNVBAoTDFN3aXNzU2lnbiBBRzEfMB0GA1UEAxMWU3dpc3NTaWduIEdvbGQgQ0EgLSBH MjAeFw0xNDA5MTkxNDEwMjVaFw0yOTA5MTUxNDEwMjVaMFQxCzAJBgNVBAYTAkNIMRUwEwYDVQQK EwxTd2lzc1NpZ24gQUcxLjAsBgNVBAMTJVN3aXNzU2lnbiBQZXJzb25hbCBHb2xkIENBIDIwMTQg LSBHMjIwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCet/RPYkWzHg09v5NQEic+C/2d TIX10wmHuxAvrofJusHZ16+o1zX4ar58VPi18Fr2Fsj7GpFGoI7QuhA4ne/dcyU7NDfNMv5vfCAO +z93y1nstYY5JgXjzT+svVpoEhAWWsTrYYNC9yaQf8LxDGxJL5P6BrYVCEL8Gv9u2941w/pzPztb KyEXwhT/uw4piqHejmZvEjrjVHW7f7NqIarwD3rbSlKhbdv0/Ux/m3jPnWJ+pv79Cxi0O3rR3D6Z URLqyTxyX1VEt6Ny6JcXJUlqPtjIhOKivfdu7OF4e/Nl3FpBKB6rSYiu+Zeq2mVmipdDXaKtHvGG 5//D1S0UFuoLAgMBAAGjggKUMIICkDAOBgNVHQ8BAf8EBAMCAQYwEgYDVR0TAQH/BAgwBgEB/wIB ADAdBgNVHQ4EFgQU2jL5SfhRzJhxZgzZzrbbkj8JS+8wHwYDVR0jBBgwFoAUWyV7lqRlUX64OfPA eGZe6Drn8O4wgf8GA1UdHwSB9zCB9DBHoEWgQ4ZBaHR0cDovL2NybC5zd2lzc3NpZ24ubmV0LzVC MjU3Qjk2QTQ2NTUxN0VCODM5RjNDMDc4NjY1RUU4M0FFN0YwRUUwgaiggaWggaKGgZ9sZGFwOi8v ZGlyZWN0b3J5LnN3aXNzc2lnbi5uZXQvQ049NUIyNTdCOTZBNDY1NTE3RUI4MzlGM0MwNzg2NjVF RTgzQUU3RjBFRSUyQ089U3dpc3NTaWduJTJDQz1DSD9jZXJ0aWZpY2F0ZVJldm9jYXRpb25MaXN0 P2Jhc2U/b2JqZWN0Q2xhc3M9Y1JMRGlzdHJpYnV0aW9uUG9pbnQwXwYDVR0gBFgwVjBUBglghXQB WQECAQYwRzBFBggrBgEFBQcCARY5aHR0cDovL3JlcG9zaXRvcnkuc3dpc3NzaWduLmNvbS9Td2lz c1NpZ24tR29sZC1DUC1DUFMucGRmMIHGBggrBgEFBQcBAQSBuTCBtjBkBggrBgEFBQcwAoZYaHR0 cDovL3N3aXNzc2lnbi5uZXQvY2dpLWJpbi9hdXRob3JpdHkvZG93bmxvYWQvNUIyNTdCOTZBNDY1 NTE3RUI4MzlGM0MwNzg2NjVFRTgzQUU3RjBFRTBOBggrBgEFBQcwAYZCaHR0cDovL29jc3Auc3dp c3NzaWduLm5ldC81QjI1N0I5NkE0NjU1MTdFQjgzOUYzQzA3ODY2NUVFODNBRTdGMEVFMA0GCSqG SIb3DQEBCwUAA4ICAQCt69uoI48fq9rTHigRvhS+LJcnMiiSXkyNEAOUYwMVdoqbnaZT4hmUn0+N yoGIReGOMPWhHy5XlpVFZcf5q4jomBBnPkv67ZrfT1sO7dtyKd4/jm9n6cC6gO+6WbHaeqOWgI62 6tqSzvbvD+ccKjPvbd5gfjp+D/+r9rlCc6gF75YiJlZPvVNzJxfiGJVw/BWoHne0v0gzap7ZtqWc KlkqitlZUdBdhZI7kwgFrD+Iw7AIKl+UNFTvlOu4c+N8TgG7hT02bc0G3fNxznKRxDxlsnORYudK LYtjyvvVSVzePGrJY5lBuBLXjH8HNVCPhGT35fwqi8amigAVUgFH9t1T/PYh1mUQ2Cg82kmiS4Qf k5LpRlLI2aGFeMjZAZef3Ub5BXj76St5XPqF5U6EM+A6g59Fctgb07/3Co8y2GZRCBxOvKannts+ kzTMhikfFwp/GGhimtB07cu5TUXbGRkTa8UtZUH26AfNd3UAMgqusaAlc98yW5dP3ArwHNNwjlw6 qERK3gDCrmr1+aJbc5QBE4vQJrRn+xFBy+8n97CVm8t2zKZx+NXom03v8gGAgngP8mzgNFYLAC/W OEGIQaxECWGImZ6n/9TUcRpP2it52309hLWIpuRwy4MS0+5hetD7w5CYwTAxs4VuhkwxM5U056cs f4SIboFSejBPCqqO0DCCB0owggYyoAMCAQICFDNloxCXhgHZ2F0RJhXzjQQxsg+6MA0GCSqGSIb3 DQEBCwUAMFQxCzAJBgNVBAYTAkNIMRUwEwYDVQQKEwxTd2lzc1NpZ24gQUcxLjAsBgNVBAMTJVN3 aXNzU2lnbiBQZXJzb25hbCBHb2xkIENBIDIwMTQgLSBHMjIwHhcNMjAwMzIwMTQ0NzIxWhcNMjEw MzIwMTQ0NzIxWjBzMQswCQYDVQQGEwJERTEPMA0GA1UECBMGQmF5ZXJuMRgwFgYDVQQKEw9TZWxi c3RkZW5rZXIgQUcxIzAhBgkqhkiG9w0BCQEWFG1haWtAc2VsYnN0ZGVua2VyLmFnMRQwEgYDVQQD EwtNYWlrIE11c2FsbDCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAK8ihbzARUg6A1WT 77ncVMCMp+1aqAVSk8hPyFrTq1KQw06EoUAmdjFsxudN4VTHbacAp77nOI2QavuJ5S159TNbtyBW /YcEVklQmJf2A40abuuJM5AZLhHBmjF3yzckZWk+nQqCsakbfD7knjxrZj8wqIzTwnfftmaBzqxX SjDDT26gA4S0zYowDKechMOsz5Gh/p2Uq6CR99n6DGu7Qj1ZPTGfI+hrDbJfbGVocdrAOtUK5Vaa AxFqZTHwAP6fiLwGfwdTjD2Mv5Fb1Wp+NegKwAxNInIjW9BoPPkeKoUlun0Qu+syXIG5wZIuGQit zVZ6ty0DZ7Qww6MYezZ9R7FdcqpV0RIeKpA4z8Edi8KSVr9crtvomcVVvlaJHPB/QzvUI69c0Mjf 4e5nH+bKw5TDqReky/yCGI3QFhjxcF5eQkyibCEDY+H+E04+e6Wout/ybC2pPvDm+8JSIfH57UXw RMsw+KJh5fvwjcbD72VJIq2ygLc9hCgEWib1OWTD/nn3i3GciAhHI90fLOAYOaAF6sZ6zyEaKegh f9iFvxPegX5c5CI3YZkNQ45NLYS1wKS8IAK+8uVm8B/4vMKmOSHBX9GvlQJ8ACDtjisXaM8ElLY7 e7ExL4xGiBhZy63SCSZgnYZ3MWbA8QzW38JEa/i1KMt+cUCjWrzvaeMGE/JRAgMBAAGjggLzMIIC 7zAfBgNVHREEGDAWgRRtYWlrQHNlbGJzdGRlbmtlci5hZzAOBgNVHQ8BAf8EBAMCA/gwNQYDVR0l BC4wLAYIKwYBBQUHAwIGCCsGAQUFBwMEBgorBgEEAYI3CgMEBgorBgEEAYI3FAICMB0GA1UdDgQW BBTSGjJ8ACOy2OebhL3ieB68ZTIT6jAfBgNVHSMEGDAWgBTaMvlJ+FHMmHFmDNnOttuSPwlL7zCB /wYDVR0fBIH3MIH0MEegRaBDhkFodHRwOi8vY3JsLnN3aXNzc2lnbi5uZXQvREEzMkY5NDlGODUx Q0M5ODcxNjYwQ0Q5Q0VCNkRCOTIzRjA5NEJFRjCBqKCBpaCBooaBn2xkYXA6Ly9kaXJlY3Rvcnku c3dpc3NzaWduLm5ldC9DTj1EQTMyRjk0OUY4NTFDQzk4NzE2NjBDRDlDRUI2REI5MjNGMDk0QkVG JTJDTz1Td2lzc1NpZ24lMkNDPUNIP2NlcnRpZmljYXRlUmV2b2NhdGlvbkxpc3Q/YmFzZT9vYmpl Y3RDbGFzcz1jUkxEaXN0cmlidXRpb25Qb2ludDBpBgNVHSAEYjBgMFQGCWCFdAFZAQIBDTBHMEUG CCsGAQUFBwIBFjlodHRwOi8vcmVwb3NpdG9yeS5zd2lzc3NpZ24uY29tL1N3aXNzU2lnbi1Hb2xk LUNQLUNQUy5wZGYwCAYGBACPegEBMIHXBggrBgEFBQcBAQSByjCBxzBkBggrBgEFBQcwAoZYaHR0 cDovL3N3aXNzc2lnbi5uZXQvY2dpLWJpbi9hdXRob3JpdHkvZG93bmxvYWQvREEzMkY5NDlGODUx Q0M5ODcxNjYwQ0Q5Q0VCNkRCOTIzRjA5NEJFRjBfBggrBgEFBQcwAYZTaHR0cDovL2dvbGQtcGVy c29uYWwtZzIub2NzcC5zd2lzc3NpZ24ubmV0L0RBMzJGOTQ5Rjg1MUNDOTg3MTY2MENEOUNFQjZE QjkyM0YwOTRCRUYwDQYJKoZIhvcNAQELBQADggEBAHf5MFhf+jGhDeA4o0QwckRPVIW9pVfNa0Kl 1H5Rvlh0kriBXQ7DVmuFHm9oM2WMWY/q/X5BE+GLtJlTB/GleWKw87gqFQQu5PozNQdPMqr5f0iw p+uBb2BSwJJ96EueKlegR9ZEkgTuPZBRkJl43x+0cOB1/oEQGHbLl5EEIcWUQ2uG3yawr80EsqrL OoZGFplTLlhwX/iQl05Tj+xOzVxDiaTOB3My/pg/d1SpqNWAAWDGvcSGps+ICR95WzurUV2N7njy 1TMwilLOsSL5M4JvYo1jBcYi7f2/rfplH4M7kXZ5LP8prkiqP4wF7KdGXH45HPlrIJyFkdqVfpTQ JJIxggQAMIID/AIBATBsMFQxCzAJBgNVBAYTAkNIMRUwEwYDVQQKEwxTd2lzc1NpZ24gQUcxLjAs BgNVBAMTJVN3aXNzU2lnbiBQZXJzb25hbCBHb2xkIENBIDIwMTQgLSBHMjICFDNloxCXhgHZ2F0R JhXzjQQxsg+6MA0GCWCGSAFlAwQCAQUAoIIBZTAYBgkqhkiG9w0BCQMxCwYJKoZIhvcNAQcBMBwG CSqGSIb3DQEJBTEPFw0yMTAyMDkxNjAzMTJaMC8GCSqGSIb3DQEJBDEiBCC544oSs1Rt/TSC09H5 LgQArLe7ou5sfeSUdpzHqnHS1DB7BgkrBgEEAYI3EAQxbjBsMFQxCzAJBgNVBAYTAkNIMRUwEwYD VQQKEwxTd2lzc1NpZ24gQUcxLjAsBgNVBAMTJVN3aXNzU2lnbiBQZXJzb25hbCBHb2xkIENBIDIw MTQgLSBHMjICFDNloxCXhgHZ2F0RJhXzjQQxsg+6MH0GCyqGSIb3DQEJEAILMW6gbDBUMQswCQYD VQQGEwJDSDEVMBMGA1UEChMMU3dpc3NTaWduIEFHMS4wLAYDVQQDEyVTd2lzc1NpZ24gUGVyc29u YWwgR29sZCBDQSAyMDE0IC0gRzIyAhQzZaMQl4YB2dhdESYV840EMbIPujANBgkqhkiG9w0BAQsF AASCAgAecMR4dmCJS6TSY/LELZ9yEKAZG/3KPeCBPzimqZ9IjQGURzorNU0KIOGDaltOC3w1wnvw QpG2I3+Inj9knu/JcKZvIvgV56WqovBfStdzxEqa/n1H/Pjx4htj5GoaHIwLXyr8T9xQPhQLroaZ KvuENVXwh2ZrbT6IoSnXsq+32aACfrcoJ9PO5UeGxKDUC5T3IGjWWwIXO4jSAnvYylHYb6OG0mjK 1yyqPQFA+QboVA3/ghw7qWVa0VwweqNkK0PXUsgcs7/IDafVeTkw6/i9udab+RplmDhmwa3jcP/v LHqOxpnEsGsLXoxroEDkQt0dLdOJ0onFi+r/7X2Id06ucacLFFiVkqe5unJed3HTiQ+K3nv09Ine QcR49SC4U3m0a4d5r5IdSU1pahB5BL+qwR8WRPoFquBJkkr6SPr5kTEiM/nw3LNyrvHPNFJhMXr8 gGY5cr8VtpwIle9Pw9oNWacoGdjT/yRVS4kh5wV0Obt3ARlJJJAiO+2n9dfcewmGvZ3VJcSwN0Xk HA+PkUG2uU8RaZ4RKrDUBkPocFJaGoqLDxpzLEs72DDG3sSBGsU+kLxMcLj4MICigZNxPt1UZ3So p1SMXTj/6D6iL49nIJDP+PuyOlH1VH2z62aD5oeskdo7CtNU5v+Mbsg/9lECSLRJigcpzNhhu5FB R2WLUgAAAAAAAA== --Apple-Mail=_7CA46C54-5EE9-4849-8854-4011F8F05925--