From MAILER-DAEMON Tue Nov 08 09:58:49 2022 Received: from list by lists.gnu.org with archive (Exim 4.90_1) id 1osQ3t-0002qY-4b for mharc-gforth@gnu.org; Tue, 08 Nov 2022 09:58:49 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1osQ3s-0002qJ-2T for gforth@gnu.org; Tue, 08 Nov 2022 09:58:48 -0500 Received: from mail.blazebone.com ([45.32.235.173]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_CHACHA20_POLY1305:256) (Exim 4.90_1) (envelope-from ) id 1osQ3m-0001cm-Sq for gforth@gnu.org; Tue, 08 Nov 2022 09:58:47 -0500 Received: from forest.lan (46-23-92-148.openbsd.amsterdam [46.23.92.148]) by aries.vpn (OpenSMTPD) with ESMTPSA id 6e1dd2ed (TLSv1.3:AEAD-CHACHA20-POLY1305-SHA256:256:NO) for ; Tue, 8 Nov 2022 15:58:36 +0100 (CET) Received: from localhost (forest.lan [local]) by forest.lan (OpenSMTPD) with ESMTPA id 10e22999 for ; Tue, 8 Nov 2022 15:58:32 +0100 (CET) User-agent: mu4e 1.8.11; emacs 28.2 From: nature To: gforth@gnu.org Subject: Memory leak in unix/pthread.fs Date: Tue, 08 Nov 2022 15:38:54 +0100 Message-ID: <87tu39ebmv.fsf@forest.lan> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" Received-SPF: pass client-ip=45.32.235.173; envelope-from=nature@blazebone.com; helo=mail.blazebone.com X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, UNPARSEABLE_RELAY=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: gforth@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gforth discussion and announcements List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Nov 2022 14:58:48 -0000 --=-=-= Content-Type: text/plain Hi, I am working on a web server written in pure Forth and I am using gforth as the implementation I develop on. Serving multiple requests without blocking requires me to venture into the realm of multi-threading and gforth's experimental support for pthread has been a blessing. It may be good for me to mention that I work on UNIX (OpenBSD). Now instead of doing like in cilk.fs and spawning a few workers and offloading requests to them, I prefer to create a separate thread each time a request comes and this led me to notice that my server wasn't able to serve more than ~3k requests before hanging and also the memory usage of my process increased a lot with each new request comming in. I then proceeded to hunt down memory leaks due to the S" S\" and S+ words with minimal impact. That is what lead me to realize that creating new threads with the PASS word was creating non-detached thread which never freed the memory once the thread exited. I am attaching a patch for unix/pthread.fs which fixed the big memory leak for me, my server is no able to server well above 3k requests without hanging/crashing (I tested 10 without any issues). --=-=-= Content-Type: text/x-patch Content-Disposition: inline; filename=pthread.fs.patch Content-Description: Patch for unix/pthread.fs --- pthread.fs.old Tue Nov 8 15:22:26 2022 +++ pthread.fs Tue Nov 8 15:22:56 2022 @@ -247,7 +247,7 @@ r> swap >r save-task r@ 's ! 1+ dup cells negate sp0 r@ 's @ -rot sp0 r@ 's +! sp0 r@ 's @ swap 0 ?DO tuck ! cell+ LOOP drop - pthread-id r@ 's 0 thread_start r> pthread_create drop ; compile-only + pthread-id r@ 's pthread_detach_attr thread_start r> pthread_create drop ; compile-only : pass ( x1 .. xn n task -- ) \ gforth-experimental \G activates task, and passes n parameters from the data stack --=-=-= Content-Type: text/plain Also optionally I am attaching another patch for cilk.fs just to correctly get the number of cores on OpenBSD. --=-=-= Content-Type: text/x-patch Content-Disposition: inline; filename=cilk.fs.patch Content-Description: Patch for cilk.fs --- cilk.fs.old Tue Nov 8 15:31:33 2022 +++ cilk.fs Tue Nov 8 15:32:43 2022 @@ -20,7 +20,7 @@ require unix/pthread.fs -e? os-type s" darwin" string-prefix? [IF] s" sysctl -n hw.ncpu" +e? os-type 2dup s" darwin" string-prefix? -rot s" openbsd" string-prefix? or [IF] s" sysctl -n hw.ncpu" [ELSE] s" nproc" [THEN] r/o open-pipe throw slurp-fid s>number drop 1 max Value cores --=-=-= Content-Type: text/plain Thank you for the great work on gforth, I can't wait for the official 1.0 to release so that OpenBSD ports won't be stuck with 0.7.3 forever! :D Cheers, -- nature --=-=-=-- From MAILER-DAEMON Tue Nov 08 10:04:39 2022 Received: from list by lists.gnu.org with archive (Exim 4.90_1) id 1osQ9W-0005Bl-Kw for mharc-gforth@gnu.org; Tue, 08 Nov 2022 10:04:38 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1osQ9U-00058k-V7 for gforth@gnu.org; Tue, 08 Nov 2022 10:04:36 -0500 Received: from mail.blazebone.com ([45.32.235.173]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_CHACHA20_POLY1305:256) (Exim 4.90_1) (envelope-from ) id 1osQ9S-0005VA-7S for gforth@gnu.org; Tue, 08 Nov 2022 10:04:36 -0500 Received: from forest.lan (46-23-92-148.openbsd.amsterdam [46.23.92.148]) by aries.vpn (OpenSMTPD) with ESMTPSA id 11df3ad1 (TLSv1.3:AEAD-CHACHA20-POLY1305-SHA256:256:NO) for ; Tue, 8 Nov 2022 16:04:29 +0100 (CET) Received: from localhost (forest.lan [local]) by forest.lan (OpenSMTPD) with ESMTPA id 0089b157 for ; Tue, 8 Nov 2022 16:04:26 +0100 (CET) References: <87ilm44zkn.fsf@forest.vpn> <87edwrk4xj.fsf@logand.com> <87ilm31nl5.fsf@forest.vpn> User-agent: mu4e 1.8.11; emacs 28.2 From: nature To: gforth@gnu.org Subject: Re: Issues with socket.fs on OpenBSD Date: Tue, 08 Nov 2022 16:00:02 +0100 In-reply-to: <87ilm31nl5.fsf@forest.vpn> Message-ID: <87pmdxebd1.fsf@forest.lan> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" Received-SPF: pass client-ip=45.32.235.173; envelope-from=nature@blazebone.com; helo=mail.blazebone.com X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, UNPARSEABLE_RELAY=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: gforth@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gforth discussion and announcements List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Nov 2022 15:04:37 -0000 --=-=-= Content-Type: text/plain After poking around some more I realized that changing the MSG_WAITALL flag used for recv to 0 (no flags) gave recv the same behavior as read and henced fixed the read-socket word on OpenBSD. Please see the attached patch. --=-=-= Content-Type: text/x-patch Content-Disposition: inline; filename=socket.fs.patch Content-Description: Patch for unix/socket.fs --- socket.fs.old Tue Nov 8 15:34:30 2022 +++ socket.fs Tue Nov 8 15:35:21 2022 @@ -476,7 +476,7 @@ : (rs) ( socket c-addr maxlen -- c-addr size ) third dup >r false blocking-mode rot fileno -rot - over >r msg_waitall recv + over >r 0 recv dup 0< IF 0 max errno dup 0<> swap ewouldblock <> and ?ior THEN --=-=-= Content-Type: text/plain Best, -- nature --=-=-=-- From MAILER-DAEMON Sat Nov 12 13:11:31 2022 Received: from list by lists.gnu.org with archive (Exim 4.90_1) id 1otuyY-00066i-S6 for mharc-gforth@gnu.org; Sat, 12 Nov 2022 13:11:30 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1otuyU-00066E-6E for gforth@gnu.org; Sat, 12 Nov 2022 13:11:26 -0500 Received: from rei.gormotte.info ([185.161.45.250]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1otuyS-00082U-2s for gforth@gnu.org; Sat, 12 Nov 2022 13:11:25 -0500 Received: by rei.gormotte.info (Postfix, from userid 5000) id 77EA11401C2; Sat, 12 Nov 2022 19:11:20 +0100 (CET) Received: from mail.aida-sea.com (localhost [127.0.0.1]) (Authenticated sender: paul.ortais@aida-sea.com) by rei.gormotte.info (Postfix) with ESMTPA id B6888140173; Sat, 12 Nov 2022 19:11:19 +0100 (CET) MIME-Version: 1.0 Date: Sat, 12 Nov 2022 19:11:19 +0100 From: paul.ortais@aida-sea.com To: nature Cc: gforth@gnu.org Subject: Re: Memory leak in unix/pthread.fs In-Reply-To: <87tu39ebmv.fsf@forest.lan> References: <87tu39ebmv.fsf@forest.lan> Message-ID: <08eaa788796f935663c47056c3d7022a@aida-sea.com> X-Sender: paul.ortais@aida-sea.com Received-SPF: pass client-ip=185.161.45.250; envelope-from=paul.ortais@aida-sea.com; helo=rei.gormotte.info X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, HTML_MESSAGE=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: gforth@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gforth discussion and announcements List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 Nov 2022 18:11:28 -0000 Great contribution, thank you very much ZP Le 2022-11-08 15:38, nature a écrit : > Hi, > > I am working on a web server written in pure Forth and I am using > gforth > as the implementation I develop on. Serving multiple requests without > blocking requires me to venture into the realm of multi-threading and > gforth's experimental support for pthread has been a blessing. It may > be good for me to mention that I work on UNIX (OpenBSD). > > Now instead of doing like in cilk.fs and spawning a few workers and > offloading requests to them, I prefer to create a separate thread each > time a request comes and this led me to notice that my server wasn't > able to serve more than ~3k requests before hanging and also the memory > usage of my process increased a lot with each new request comming in. > I > then proceeded to hunt down memory leaks due to the S" S\" and S+ words > with minimal impact. That is what lead me to realize that creating new > threads with the PASS word was creating non-detached thread which never > freed the memory once the thread exited. > > I am attaching a patch for unix/pthread.fs which fixed the big memory > leak for me, my server is no able to server well above 3k requests > without hanging/crashing (I tested 10 without any issues). > > Also optionally I am attaching another patch for cilk.fs just to > correctly get the number of cores on OpenBSD. > > Thank you for the great work on gforth, I can't wait for the official > 1.0 to release so that OpenBSD ports won't be stuck with 0.7.3 forever! > :D > > Cheers, From MAILER-DAEMON Wed Nov 23 21:30:05 2022 Received: from list by lists.gnu.org with archive (Exim 4.90_1) id 1oy205-0001y0-Nw for mharc-gforth@gnu.org; Wed, 23 Nov 2022 21:30:05 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1oy1dU-0006q5-GV for gforth@gnu.org; Wed, 23 Nov 2022 21:06:44 -0500 Received: from mail-ot1-x32b.google.com ([2607:f8b0:4864:20::32b]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1oy1dS-0000eX-OQ for gforth@gnu.org; Wed, 23 Nov 2022 21:06:44 -0500 Received: by mail-ot1-x32b.google.com with SMTP id t19-20020a9d7753000000b0066d77a3d474so146502otl.10 for ; Wed, 23 Nov 2022 18:06:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=to:subject:message-id:date:from:mime-version:from:to:cc:subject :date:message-id:reply-to; bh=fUyVzO3a1m1YQfXo62LUi2dddAzXsa19cN9+/qeHHcc=; b=icesGkoszyfAd1pjiyf9kGTNnkzKMQK1he0il/60zk9GFv2+FGdXLEfjWyVv7uxfli 34OSLQAHXqyk23wpy9XWyV9luQtAAtlBCfwPUVrgBqKpx45/VpNj8cX8zduR/oRO0+En cde9AYl/eyY8H5GlC1DjvtwA0KfGF5WsTqoWKqVrc0JukhRZhnr/Zx/U7tkzPCjZH95Q fG03XBkS3b7ZS+d048scEeHoa0QZwUWhqxFQaKvja1/KyXx+e0av00AbaEHEaH9eUGyP uhDCAZUPNaeVgljDNzQDyu28EpHFSYk7DFg1m+gFEhxGXjn5JYk1ekpiDlzhNmfdCa5M steg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=fUyVzO3a1m1YQfXo62LUi2dddAzXsa19cN9+/qeHHcc=; b=30S/DA1Dfh0Za8Wv1KzalgMIwAoijNAVzzDOziYCi50kWG2MGeXrJBllevP0wy4HBO MpUFEyeQlXpuFdh9e8EJ3M29tZNcG1zjOTNgP+KH2m4/WTXy6s2Iqxy5ZywaoOjRk6OK OYCwc9gv99aNLbm7xExnUV96bYXeTIJaHR8pBlOF5vj45M3SmLn4yE1vEtIj9FV5EcJx RWDmTGgsMTk0QEfFYFNPG4dV+qzhBzoLTxqMEdMdeLpkWXFOu3OK+VyJZF4uxrvD/SfQ nplMSQpekyzYnGkfH09klzuZzew/dawF4JNjVaRxcrUyqYlOzrUHCAHh/KKSBa8FRv+d G7gw== X-Gm-Message-State: ANoB5pnjyc5e9x9Ug7O2oXSzznpbf1ky92qtbvnV0obZAkLtyYUPyxYS 5cnzU47HBJnuJU7jjhey25bIClnl6uT+zO7UMU/6vFO2sFw= X-Google-Smtp-Source: AA0mqf4HS6NY23XgRd0rGmxIy0Tn3DFh6Z5cFzaOGH5Aht2x15o7INh4O9ITcZd3XBKteBkbpnWHoTbaE09WOCBx3LI= X-Received: by 2002:a9d:7092:0:b0:660:d7e8:b212 with SMTP id l18-20020a9d7092000000b00660d7e8b212mr7025109otj.226.1669255600331; Wed, 23 Nov 2022 18:06:40 -0800 (PST) MIME-Version: 1.0 From: Wolfgang Daum Date: Wed, 23 Nov 2022 21:06:29 -0500 Message-ID: Subject: GFORTH on M68k 68030 System To: gforth@gnu.org Received-SPF: pass client-ip=2607:f8b0:4864:20::32b; envelope-from=ws5daum@gmail.com; helo=mail-ot1-x32b.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-Mailman-Approved-At: Wed, 23 Nov 2022 21:30:02 -0500 Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: gforth@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gforth discussion and announcements List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Nov 2022 02:06:44 -0000 Has anyone been successful in porting GFORTH to a m68k 68030 SBC? The archive has only 3 messages on this topic. I already have a m68k-elf cross compiler installed. Any inputs would be appreciated. Best wdaum From MAILER-DAEMON Thu Nov 24 17:53:44 2022 Received: from list by lists.gnu.org with archive (Exim 4.90_1) id 1oyL6G-0001Tz-Cd for mharc-gforth@gnu.org; Thu, 24 Nov 2022 17:53:44 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1oyL6E-0001Tj-PG for gforth@gnu.org; Thu, 24 Nov 2022 17:53:42 -0500 Received: from a4.complang.tuwien.ac.at ([128.130.173.65]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1oyL6C-0003BR-6a for gforth@gnu.org; Thu, 24 Nov 2022 17:53:41 -0500 Received: from anton by a4.complang.tuwien.ac.at with local (Exim 4.94.2) (envelope-from ) id 1oyL65-00D6KU-MO; Thu, 24 Nov 2022 23:53:33 +0100 Date: Thu, 24 Nov 2022 23:53:33 +0100 From: Anton Ertl To: Wolfgang Daum Cc: gforth@gnu.org Subject: Re: GFORTH on M68k 68030 System Message-ID: Reply-To: anton@mips.complang.tuwien.ac.at References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Received-SPF: none client-ip=128.130.173.65; envelope-from=anton@a4.complang.tuwien.ac.at; helo=a4.complang.tuwien.ac.at X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_NONE=0.001, SPF_NONE=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: gforth@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gforth discussion and announcements List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Nov 2022 22:53:43 -0000 On Wed, Nov 23, 2022 at 09:06:29PM -0500, Wolfgang Daum wrote: > Has anyone been successful in porting GFORTH to a m68k 68030 SBC? The > archive has only 3 messages on this topic. > > I already have a m68k-elf cross compiler installed. > > Any inputs would be appreciated. Gforth is pretty portable. Unfortunately I never came across a 68k machine for testing, but it should work, either with the m68k host (so apparently someone tested after all), or the generic host (the latter should work everywhere; if all else fails, configure with no_dynamic_default=1). Cross-compiling is a hurdle. We have done some preparations for it, and you can read about them in INSTALL and INSTALL.md. But it's still not mainstream, and you may encounter some problems. But I guess you are used to that:-). - anton From MAILER-DAEMON Thu Nov 24 21:03:36 2022 Received: from list by lists.gnu.org with archive (Exim 4.90_1) id 1oyO40-0000py-KV for mharc-gforth@gnu.org; Thu, 24 Nov 2022 21:03:36 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1oyO3y-0000pg-Mx for gforth@gnu.org; Thu, 24 Nov 2022 21:03:34 -0500 Received: from logand.com ([37.48.87.44]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1oyO3w-000883-QC for gforth@gnu.org; Thu, 24 Nov 2022 21:03:34 -0500 Received: by logand.com (Postfix, from userid 1001) id AC74A19E72B; Fri, 25 Nov 2022 03:03:20 +0100 (CET) X-Mailer: emacs 28.1 (via feedmail 11-beta-1 I) From: Tomas Hlavaty To: nature , gforth@gnu.org Subject: Re: Memory leak in unix/pthread.fs In-Reply-To: <87tu39ebmv.fsf@forest.lan> References: <87tu39ebmv.fsf@forest.lan> Date: Fri, 25 Nov 2022 03:03:17 +0100 Message-ID: <87cz9boknu.fsf@logand.com> MIME-Version: 1.0 Content-Type: text/plain Received-SPF: pass client-ip=37.48.87.44; envelope-from=tom@logand.com; helo=logand.com X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: gforth@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gforth discussion and announcements List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Nov 2022 02:03:34 -0000 On Tue 08 Nov 2022 at 15:38, nature wrote: > Serving multiple requests without > blocking requires me to venture into the realm of multi-threading another option is fork > I prefer to create a separate thread each > time a request comes and this led me to notice that my server wasn't > able to serve more than ~3k requests before hanging and also the memory > usage of my process increased a lot with each new request comming in. I > then proceeded to hunt down memory leaks due to the S" S\" and S+ words > with minimal impact. That is what lead me to realize that creating new > threads with the PASS word was creating non-detached thread which never > freed the memory once the thread exited. > > I am attaching a patch for unix/pthread.fs which fixed the big memory > leak for me, my server is no able to server well above 3k requests > without hanging/crashing (I tested 10 without any issues). That is interesting. Even with your patch, I still cannot create more than cca 500 threads. There seem to be other resources like pipes probably hitting an ulimit. It also seems that those do not get disposed properly, do they? How do you dispose a thread properly in gforth? How do you create a thread in your case? Do you use the gforth default code which creates pipes? Do you increase ulimit? Would it be possible to share your example? So far, I have only seen forth examples with fixed number of threads. Here is my test case: --- task3.fs \ pthreads per request \ (compile "gforth task3.fs") require ~/mr/gforth/unix/pthread.fs : seconds 1000 * ms ; : task1 ( n -- ) dup . 10 seconds . ; : mktask stacksize4 newtask4 ; \ fix thread leak id:87tu39ebmv.fsf@forest.lan \ without this, VIRT (and RES?) stays proportinal to #threads \ with this, VIRT and RES goes down but not enough : (pass) ( x1 .. xn n task -- ) \ gforth-experimental r> swap >r save-task r@ 's ! 1+ dup cells negate sp0 r@ 's @ -rot sp0 r@ 's +! sp0 r@ 's @ swap 0 ?DO tuck ! cell+ LOOP drop pthread-id r@ 's pthread_detach_attr thread_start r> pthread_create drop ; compile-only : pass ( x1 .. xn n task -- ) \ gforth-experimental \G activates task, and passes n parameters from the data stack ]] (pass) up! sp0 ! thread-init [[ ; immediate compile-only : task& ( n -- ) 1 mktask pass task1 ; : test ( n -- ) 0 do i task& loop ; \ 500 test 1000 test --- error task3.fs:27:6: error: Invalid memory address 1000 >>>test<<< Backtrace: /nix/store/hj2683h26frrj7wckziclkiryacq539b-gforth-0.7.9_20220811/share/gforth/0.7.9_20220811/libcc.fs:931:14: 0 $7FD2E1CFED00 call-c ~/mr/gforth/unix/pthread.fs:224:18: 1 $7FD2E1D25BD0 create_pipe 2 $7FD16FF56000 task3.fs:9:21: 3 $7FD2E1D28930 newtask4 task3.fs:23:20: 4 $7FD2E1D28C48 mktask task3.fs:24:24: 5 $7FD2E1D28CF8 task& 6 $1FD 7 $3E8 From MAILER-DAEMON Thu Nov 24 21:14:09 2022 Received: from list by lists.gnu.org with archive (Exim 4.90_1) id 1oyOED-0005A2-47 for mharc-gforth@gnu.org; Thu, 24 Nov 2022 21:14:09 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1oyOEB-00059k-GU for gforth@gnu.org; Thu, 24 Nov 2022 21:14:07 -0500 Received: from logand.com ([37.48.87.44]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1oyOEA-0001KI-2m for gforth@gnu.org; Thu, 24 Nov 2022 21:14:07 -0500 Received: by logand.com (Postfix, from userid 1001) id 50F0219E72B; Fri, 25 Nov 2022 03:14:04 +0100 (CET) X-Mailer: emacs 28.1 (via feedmail 11-beta-1 I) From: Tomas Hlavaty To: nature , gforth@gnu.org Subject: Re: Memory leak in unix/pthread.fs In-Reply-To: <87cz9boknu.fsf@logand.com> References: <87tu39ebmv.fsf@forest.lan> <87cz9boknu.fsf@logand.com> Date: Fri, 25 Nov 2022 03:14:02 +0100 Message-ID: <87a64fok5x.fsf@logand.com> MIME-Version: 1.0 Content-Type: text/plain Received-SPF: pass client-ip=37.48.87.44; envelope-from=tom@logand.com; helo=logand.com X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: gforth@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gforth discussion and announcements List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Nov 2022 02:14:07 -0000 On Fri 25 Nov 2022 at 03:03, Tomas Hlavaty wrote: > It also seems that those do not get disposed properly, do they? The pipes do get closed, except one. I think it is done by the :noname kill-task in newtask4.