From: Benno Senoner <sbe...@gardena.net> Subject: Scheduling latencies news: less RAM = less latency Date: 1999/07/31 Message-ID: <fa.k412ekv.dmuigt@ifi.uio.no>#1/1 X-Deja-AN: 507397611 Original-Date: Sat, 31 Jul 1999 23:34:10 +0200 Sender: owner-linux-ker...@vger.rutgers.edu Content-Transfer-Encoding: 7BIT Original-Message-Id: <99073123400600.00720@linuxhost.localdomain> To: linux-ker...@vger.rutgers.edu, alsa-de...@alsa-project.org Content-Type: text/plain; charset=US-ASCII X-Orcpt: rfc822;linux-kernel-outgoing-dig Organization: Internet mailing list MIME-Version: 1.0 Newsgroups: fa.linux.kernel X-Loop: majord...@vger.rutgers.edu Hi folks, again news as Roger Larrson suspected, there are some parts in the kernel which have a too long execution path: On his PPro with 512MB RAM, d_lookup takes up to 80ms to execute ! (I put Roger's latency profiling patch on my page) I tested this on my PII400 (UP box) + 256MB RAM: The test-kernel was 2.2.10 + a buffer.c patch (from Andrea) + uaccess.h (from Mingo) patch, which insert additional schedule() calls. I made 3 tests, booting with mem=64m , mem=128m and mem=256m the audio buffer was 4.35ms ( 3x256 bytes) the results are very interesting: (look at the diagrams !) mem=64m : /proc stress 3.5ms latency 0 overruns (this is VERY good !) disk write: 12.5ms latency 18 overruns disk copy: 11.1ms latency 14 overruns disk read: 6.3ms latency 16 overruns (quite nice :-) ) http://www.gardena.net/benno/linux/audio/2.2.10+patch+64mb/3x256.html mem=128m : /proc stress 5.4ms latency 102 overruns disk write: 14.5ms latency 20 overruns disk copy: 11.4ms latency 43 overruns disk read: 10.5ms latency 7 overruns http://www.gardena.net/benno/linux/audio/2.2.10+patch+128mb/3x256.html mem=256m : /proc stress 9.2ms latency 101 overruns disk write: 48.7ms latency 13 overruns disk copy: 31.6ms latency 24 overruns disk read: 11.9ms latency 4 overruns http://www.gardena.net/benno/linux/audio/2.2.10+patch+256mb/3x256.html IMHO having less memory , leads to less mem for disk buffering, and less pages to process, so the kernel routines take less time go through lists, inodes etc. I hope that the delay of the release of Mingo's patch is related to the fact that he tries to shorten these paths, which is not always a trivial task. Mingo, can you confirm this ? comments ? ciao, Benno. -- Benno Senoner E-Mail: sbe...@gardena.net Linux scheduling latency benchmarks http://www.gardena.net/benno/linux/audio - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.rutgers.edu Please read the FAQ at http://www.tux.org/lkml/
From: Linus Torvalds <torva...@transmeta.com> Subject: Re: Scheduling latencies news: less RAM = less latency Date: 1999/07/31 Message-ID: <fa.o7qbhnv.g4o7r9@ifi.uio.no>#1/1 X-Deja-AN: 507397612 Original-Date: Sat, 31 Jul 1999 12:59:13 -0700 (PDT) Sender: owner-linux-ker...@vger.rutgers.edu Original-Message-ID: <Pine.LNX.4.10.9907311246150.1878-100000@penguin.transmeta.com> References: <fa.k412ekv.dmuigt@ifi.uio.no> To: Benno Senoner <sbe...@gardena.net> X-Authentication-Warning: penguin.transmeta.com: torvalds owned process doing -bs Content-Type: TEXT/PLAIN; charset=US-ASCII X-Orcpt: rfc822;linux-kernel-outgoing-dig Organization: Internet mailing list MIME-Version: 1.0 Newsgroups: fa.linux.kernel X-Loop: majord...@vger.rutgers.edu On Sat, 31 Jul 1999, Benno Senoner wrote: > > as Roger Larrson suspected, there are some parts in the kernel > which have a too long execution path: > On his PPro with 512MB RAM, d_lookup takes up to 80ms to execute ! Somebody has bogus profiling information. There's no way it's going to take that long unless you have a broken filesystem that doesn't do the hashes right and hashes everything to the same entry. Even if all of memory is just dcache information, which is basically impossible to do. If you get numbers like the above, then ALL your numbers are extremely suspect, and I would seriously look into your methodology if I were you. Linus - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.rutgers.edu Please read the FAQ at http://www.tux.org/lkml/
From: Ingo Molnar <mi...@chiara.csoma.elte.hu> Subject: Re: Scheduling latencies news: less RAM = less latency Date: 1999/07/31 Message-ID: <fa.nlpl0uv.1h4iigl@ifi.uio.no>#1/1 X-Deja-AN: 507400002 Original-Date: Sat, 31 Jul 1999 22:10:03 +0200 (CEST) Sender: owner-linux-ker...@vger.rutgers.edu Original-Message-ID: <Pine.LNX.3.96.990731220052.25534F-100000@chiara.csoma.elte.hu> References: <fa.o7qbhnv.g4o7r9@ifi.uio.no> To: Linus Torvalds <torva...@transmeta.com> Content-Type: TEXT/PLAIN; charset=US-ASCII X-Orcpt: rfc822;linux-kernel-outgoing-dig Organization: Internet mailing list MIME-Version: 1.0 Newsgroups: fa.linux.kernel X-Loop: majord...@vger.rutgers.edu On Sat, 31 Jul 1999, Linus Torvalds wrote: > Somebody has bogus profiling information. There's no way it's going to no, it really happens. With 512M RAM and a 4-way Xeon i easily got 20ms+ latencies. These latencies are rare because it's caused by prune_dcache(), but they do happen. some other latencies: si_meminfo() and si_swapinfo() caused a ~5msec delay even on this box. Console switching and scrolling (when at the end of video memory) caused ~2msec latencies. big-file truncating caused up to 20msec latencies here, and i havent even tried too hard. exit() on a big process caused 8msec latency. fork() on a big process likewise. looking at a big process in /proc caused similar latencies. sync_old_buffers() introduced big latencies as well (scanning several thousands of buffers). Some 'bad' drivers like the PS2 keyboard code caused 1-3 msec latencies as well. All these things are fixed by my previous patch - but i'm still working on the dcache stuff though. the conceptual reason of these problems is that on big memory boxes there is nothing that forces processes to sleep. This means we have to put 'scheduling controls' more explicitly to various points in the kernel. I do not claim my patch is the correct approach, but something like that has to be done i'm quite sure. I've been working on this in the last 3 days, and i'm starting to reach the end of the (quite long) 'latency source' list. The box feels much more responsive interactively even under high and complex load. -- mingo - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.rutgers.edu Please read the FAQ at http://www.tux.org/lkml/
From: Linus Torvalds <torva...@transmeta.com> Subject: Re: Scheduling latencies news: less RAM = less latency Date: 1999/07/31 Message-ID: <fa.o7q3gvv.g4g6je@ifi.uio.no>#1/1 X-Deja-AN: 507405151 Original-Date: Sat, 31 Jul 1999 13:16:01 -0700 (PDT) Sender: owner-linux-ker...@vger.rutgers.edu Original-Message-ID: <Pine.LNX.4.10.9907311311100.1878-100000@penguin.transmeta.com> References: <fa.nlpl0uv.1h4iigl@ifi.uio.no> To: Ingo Molnar <mi...@chiara.csoma.elte.hu> X-Authentication-Warning: penguin.transmeta.com: torvalds owned process doing -bs Content-Type: TEXT/PLAIN; charset=US-ASCII X-Orcpt: rfc822;linux-kernel-outgoing-dig Organization: Internet mailing list MIME-Version: 1.0 Newsgroups: fa.linux.kernel X-Loop: majord...@vger.rutgers.edu On Sat, 31 Jul 1999, Ingo Molnar wrote: > > no, it really happens. With 512M RAM and a 4-way Xeon i easily got > 20ms+ latencies. These latencies are rare because it's caused by > prune_dcache(), but they do happen. prune_dcache() I can believe. But the report was about d_lookup(). So somebody is using bad profiling information, and that's dangerous. Also, the si_meminfo() etc stuff is just ridiculous. It's not a question of latency: it's a question of CPU usage. We need to just get rid of those functions instead of hacking around them - regardless of whether you add "reschedule" calls in them, they just eat too much CPU, plain and simple. Again, please don't treat the symptoms - I will not accept patches that just say "oh, this is crap, so let's reschedule a bit here". They need to be fixed properly or not at all. Linus - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.rutgers.edu Please read the FAQ at http://www.tux.org/lkml/
From: Ingo Molnar <mi...@chiara.csoma.elte.hu> Subject: Re: Scheduling latencies news: less RAM = less latency Date: 1999/07/31 Message-ID: <fa.nmpn16v.1m4gioi@ifi.uio.no>#1/1 X-Deja-AN: 507405153 Original-Date: Sat, 31 Jul 1999 22:22:57 +0200 (CEST) Sender: owner-linux-ker...@vger.rutgers.edu Original-Message-ID: <Pine.LNX.3.96.990731221743.25534I-100000@chiara.csoma.elte.hu> References: <fa.o7q3gvv.g4g6je@ifi.uio.no> To: Linus Torvalds <torva...@transmeta.com> Content-Type: TEXT/PLAIN; charset=US-ASCII X-Orcpt: rfc822;linux-kernel-outgoing-dig Organization: Internet mailing list MIME-Version: 1.0 Newsgroups: fa.linux.kernel X-Loop: majord...@vger.rutgers.edu On Sat, 31 Jul 1999, Linus Torvalds wrote: > prune_dcache() I can believe. But the report was about d_lookup(). So > somebody is using bad profiling information, and that's dangerous. sure - d_lookup() is definitely bogus. > Also, the si_meminfo() etc stuff is just ridiculous. It's not a question > of latency: it's a question of CPU usage. We need to just get rid of > those functions instead of hacking around them - regardless of whether > you add "reschedule" calls in them, they just eat too much CPU, plain > and simple. Again, please don't treat the symptoms - I will not accept > patches that just say "oh, this is crap, so let's reschedule a bit > here". They need to be fixed properly or not at all. i agree - I wanted to have something for 2.2 for people to test - i wanted to cure symptoms first, we have way too many latency sources right now and i first wanted to get something that can be tested. I think that the uaccess.h changes are too intrusive as well - i'll rework these things. Ingo - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.rutgers.edu Please read the FAQ at http://www.tux.org/lkml/
From: Benno Senoner <sbe...@gardena.net> Subject: Re: Scheduling latencies news: less RAM = less latency Date: 1999/07/31 Message-ID: <fa.k316esv.cmuiov@ifi.uio.no>#1/1 X-Deja-AN: 507412150 Original-Date: Sun, 1 Aug 1999 00:23:43 +0200 Sender: owner-linux-ker...@vger.rutgers.edu Content-Transfer-Encoding: 7BIT Original-Message-Id: <99080100513801.00720@linuxhost.localdomain> References: <fa.o7q3gvv.g4g6je@ifi.uio.no> To: torva...@transmeta.com, Ingo Molnar <mi...@chiara.csoma.elte.hu> Original-References: <Pine.LNX.4.10.9907311311100.1878-100...@penguin.transmeta.com> Content-Type: text/plain; charset=US-ASCII X-Orcpt: rfc822;linux-kernel-outgoing-dig Organization: Internet mailing list MIME-Version: 1.0 Newsgroups: fa.linux.kernel X-Loop: majord...@vger.rutgers.edu On Sat, 31 Jul 1999, Linus Torvalds wrote: > On Sat, 31 Jul 1999, Ingo Molnar wrote: > > > > no, it really happens. With 512M RAM and a 4-way Xeon i easily got > > 20ms+ latencies. These latencies are rare because it's caused by > > prune_dcache(), but they do happen. > > prune_dcache() I can believe. But the report was about d_lookup(). So > somebody is using bad profiling information, and that's dangerous. > > Also, the si_meminfo() etc stuff is just ridiculous. It's not a question > of latency: it's a question of CPU usage. We need to just get rid of those > functions instead of hacking around them - regardless of whether you add > "reschedule" calls in them, they just eat too much CPU, plain and simple. > Again, please don't treat the symptoms - I will not accept patches that > just say "oh, this is crap, so let's reschedule a bit here". They need to > be fixed properly or not at all. > > Linus Linus, I agree that this solution is perhaps not the cleanest, one thing to check, is how much slower this kernel would be compared to a standard kernel. I think "reschedules" during lenghty kernel operations is not a so bad idea, the important thing is not to reschedule too often, to avoid wasting too much CPU time in the scheduler. I'm easily willing to trade 1-5% of the CPU in exchange of a responsive <5ms latency system. If the performance drop worries you, we could add this as a compile time option, "kernel optimized for server", or "kernel optimized for multimedia" . If's ridiculous to get up to 150ms latencies on a powerful machine like the PII400 on Linux. Even a Windows user (with a properly tuned machine) laughs at these values. Simple reschedules in uaccess.h + buffer.c lowered the latency *DRAMATICALLY* on my box, about an order of magnitude. ( /proc down to 3ms , disk read down to 6ms) Linus, what are your proposal for making the kernel "low-latency" in a CLEAN way ? I think making the kernel fully preemptable is not an easy task and will not happen very soon. Plus what disturbs me is the busy-waiting for RT processes for sleeps <2ms 2ms is PLENTY of time on modern CPUs,and I call THIS wasitng CPU time, therefore we should change this approach. comments ? P.S: the profiling patch and infos are from Roger Larrson (nra02...@norran.net). The profiling-patch is on my page if you want. Benno. -- Benno Senoner E-Mail: sbe...@gardena.net Linux scheduling latency benchmarks http://www.gardena.net/benno/linux/audio - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.rutgers.edu Please read the FAQ at http://www.tux.org/lkml/
From: Ingo Molnar <mi...@chiara.csoma.elte.hu> Subject: Re: Scheduling latencies news: less RAM = less latency Date: 1999/07/31 Message-ID: <fa.nj9r2ev.1jk8i0m@ifi.uio.no>#1/1 X-Deja-AN: 507416708 Original-Date: Sat, 31 Jul 1999 23:00:08 +0200 (CEST) Sender: owner-linux-ker...@vger.rutgers.edu Original-Message-ID: <Pine.LNX.3.96.990731225242.26845A-100000@chiara.csoma.elte.hu> References: <fa.k316esv.cmuiov@ifi.uio.no> To: Benno Senoner <sbe...@gardena.net> Content-Type: TEXT/PLAIN; charset=US-ASCII X-Orcpt: rfc822;linux-kernel-outgoing-dig Organization: Internet mailing list MIME-Version: 1.0 Newsgroups: fa.linux.kernel X-Loop: majord...@vger.rutgers.edu On Sun, 1 Aug 1999, Benno Senoner wrote: > I think "reschedules" during lenghty kernel operations is not a so bad > idea, [...] i think you have a misconception - the concept of decreasing latencies in a mandatory way is nothing new to Linux, we've been doing it for quite some time. (mem.c, lp.c, etc.) We just have to export it to all affected places - now that bigger memory boxes became widespread. We already (of course!) do 'reschedules during lengthy kernel operations', when the need arises, we just dont do it explicitly when _another_ process wants us to reschedule. The patch i've sent fills this hole - more coming. > I'm easily willing to trade 1-5% of the CPU in exchange of a responsive > <5ms latency system. my guesstimate is that we will not see any numbers getting worse. Actually, some RL numbers will get much better. (eg. i can already log in much faster during heavy kernel compiles) The thing you missed i think is that we do not have to do a reschedule, we just have to check the (local) ->need_resched flag at a couple of well-chosen places. (see the patch) > If the performance drop worries you, we could add this as a compile time > option, "kernel optimized for server", or "kernel optimized for > multimedia" . you are arguing the wrong point ... Latency _is_ top priority in Linux. And it's not hard to get it, it's hard to _measure_ it reliably. (and this is why your tool is so cool.) > If's ridiculous to get up to 150ms latencies on a powerful machine like > the PII400 on Linux. i can get excellent latencies on my box, without giving up speed - good enough? -- mingo - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.rutgers.edu Please read the FAQ at http://www.tux.org/lkml/
From: Linus Torvalds <torva...@transmeta.com> Subject: Re: Scheduling latencies news: less RAM = less latency Date: 1999/08/01 Message-ID: <fa.o8a1g7v.hke7b0@ifi.uio.no>#1/1 X-Deja-AN: 507468791 Original-Date: Sat, 31 Jul 1999 17:34:32 -0700 (PDT) Sender: owner-linux-ker...@vger.rutgers.edu Original-Message-ID: <Pine.LNX.4.10.9907311721130.2036-100000@penguin.transmeta.com> References: <fa.nmpn16v.1m4gioi@ifi.uio.no> To: Ingo Molnar <mi...@chiara.csoma.elte.hu> X-Authentication-Warning: penguin.transmeta.com: torvalds owned process doing -bs Content-Type: TEXT/PLAIN; charset=US-ASCII X-Orcpt: rfc822;linux-kernel-outgoing-dig Organization: Internet mailing list MIME-Version: 1.0 Newsgroups: fa.linux.kernel X-Loop: majord...@vger.rutgers.edu On Sat, 31 Jul 1999, Ingo Molnar wrote: > > On Sat, 31 Jul 1999, Linus Torvalds wrote: > > > prune_dcache() I can believe. But the report was about d_lookup(). So > > somebody is using bad profiling information, and that's dangerous. > > sure - d_lookup() is definitely bogus. Note that I suspect that prune_dcache() being a problem spot is probably entirely valid, but for all the wrong reasons. For example, I suspect that you can find "free_inodes()" very high on your profiles if you start really looking. We have a few bad behaviour corner cases right now that aren't actually all that hard to trigger: because the new code (rightly) refuses to free inodes when there are pages associated with them (2.2.x used to just force-free the pages), the logic to handle "max_inodes" realy is fairly broken. That brokenness probably feeds over into prune_dcache(). We try desperately to free inodes, but we can't because a lot of them have pages associated with them, and that in turn makes us do bad things with the dcache too (the inode free logic thinks that the reason the inodes are locked down is a dcache reference - something that _used_ to be true). So you end up trying to prune the dcache even when the dcache really doesn't need to be pruned at all, and in situtations where it might not actually be a good idea (again, the dcache pruning refuses to prune an entry with an inode that has active pages...) I've considered just getting rid of the max_inodes thing. The only reason it exists is because we currently don't free the inodes due to memory fragmentation, so we need to artifically cut off the inode supply at some point. We might want to try just using kmalloc/kfree (or a specialized slab thing), but I suspect that will waste much more memory than we do now. It may be worth trying, though. If you're really doing tests in this area, that's something worth looking into, I suspect. I bet you'll have a much harder time triggering dcache problems. > i agree - I wanted to have something for 2.2 for people to test - i wanted > to cure symptoms first, we have way too many latency sources right now and > i first wanted to get something that can be tested. I think that the > uaccess.h changes are too intrusive as well - i'll rework these things. Note that the "si_meminfo()" thing really started its life as a debugging check: verifying that the free pages count actually matched what you got if you counted the pages by hand. It was later dis-used from that capacity, as it's not even correct: si_meminfo gets the wrong results for pages that are part of multi-page allocations (but we didn't use to support multi-page allocations originally). But in the meantime it had been hooked into sys_getinfo(), and thus it stayed on, even though it really is a completely broken piece of code. It's really ridiculous. The only thing si_meminfo really gets us is the number of shared pages, and to calculate that it often uses about 10% of the CPUtime when you have "xosview" active or something like that. We could trivially keep track of the number of shared pages on the fly, but I've been lazy. But getting rid of some of these problems might be valid on 2.2.x too. Linus - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.rutgers.edu Please read the FAQ at http://www.tux.org/lkml/
From: Linus Torvalds <torva...@transmeta.com> Subject: Re: Scheduling latencies news: less RAM = less latency Date: 1999/08/01 Message-ID: <fa.o7qbgfv.h4o736@ifi.uio.no>#1/1 X-Deja-AN: 507470963 Original-Date: Sat, 31 Jul 1999 17:44:30 -0700 (PDT) Sender: owner-linux-ker...@vger.rutgers.edu Original-Message-ID: <Pine.LNX.4.10.9907311737080.2036-100000@penguin.transmeta.com> References: <fa.k316esv.cmuiov@ifi.uio.no> To: Benno Senoner <sbe...@gardena.net> X-Authentication-Warning: penguin.transmeta.com: torvalds owned process doing -bs Content-Type: TEXT/PLAIN; charset=US-ASCII X-Orcpt: rfc822;linux-kernel-outgoing-dig Organization: Internet mailing list MIME-Version: 1.0 Newsgroups: fa.linux.kernel X-Loop: majord...@vger.rutgers.edu On Sun, 1 Aug 1999, Benno Senoner wrote: > > I think "reschedules" during lenghty kernel operations is not a so bad > idea, the important thing is not to reschedule too often, to avoid wasting > too much CPU time in the scheduler. Oh, I agree that "reschedule()" can be a very nice thing, and I'm not against adding them where necessary. The thing that makes me reall ynervous about adding them is that it's a classic case of "easy solution", and once we start going down that path code quality will start suffering if people aren't REALLY careful. I suspect that in just about _all_ of the really bad cases you can fundamentally trace the badness to Linux being stupid rather than any inherently "lengthy" operation. And then the right fix is to fix the stupidity rather than add a reschedule. There are cases where this isn't true. A classic such case is a large "read()" system call - where we really may end up doing a really lengthy operation simply because the user asks us to. That's a case where it's not about being stupid, and where a explicit re-schedule makes 100% sense. But > I'm easily willing to trade 1-5% of the CPU in exchange of a responsive <5ms > latency system. It shouldn't be necessary. In most cases bad latency should be traceable to bad performance - you can improve latency by _improving_ performance rather than making it worse. > If the performance drop worries you, we could add this as a compile time > option, "kernel optimized for server", or "kernel optimized for multimedia" . That is the Windows NT approach, and it is _wrong_. > If's ridiculous to get up to 150ms latencies on a powerful machine like the > PII400 on Linux. Agreed. And it is even more ridiculous to add stupid code to "fix" it. > Linus, what are your proposal for making the kernel "low-latency" in a CLEAN > way ? I sent out two of them. "si_meminfo()" should just go away. Same goes for "prune_dcache()", although the latter probably doesn't need to go away, it just needs to have fewer bad corner cases. > I think making the kernel fully preemptable is not an easy task and will > not happen very soon. It will not happen EVER. Pre-emptable kernels are stupid, and anybody who thinks otherwise needs to get his head examined. But trying to work around bad programming by adding MORE bad programming is also extremely stupid. Linus - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.rutgers.edu Please read the FAQ at http://www.tux.org/lkml/
From: Benno Senoner <sbe...@gardena.net> Subject: Low-latency patches working GREAT (<2.9ms audio latency), see testresults ,but ISDN troubles Date: 1999/08/28 Message-ID: <fa.k21ohcv.fm4h8u@ifi.uio.no>#1/1 X-Deja-AN: 518278955 Original-Date: Sat, 28 Aug 1999 22:40:57 +0200 Sender: owner-linux-ker...@vger.rutgers.edu Content-Transfer-Encoding: 7BIT Original-Message-Id: <99082823144300.00799@linuxhost.localdomain> To: linux-ker...@vger.rutgers.edu, linux-so...@vger.rutgers.edu, alsa-de...@alsa-project.org Content-Type: text/plain; charset=US-ASCII X-Orcpt: rfc822;linux-kernel-outgoing-dig Organization: Internet mailing list MIME-Version: 1.0 Newsgroups: fa.linux.kernel X-Loop: majord...@vger.rutgers.edu Hi, I benchmarked Mingo's latest low-latency patches (2.2.10-N6 a bit modified) The patches give me excellent results with sporadic very 2.9ms peaks ! See the testresults here: http://www.gardena.net/benno/linux/audio/2.2.10-n6b/index.html You can get the patch here: http://www.gardena.net/benno/linux/audio/patches/lowlatency-2.2.10-N6B.patch More details on my audio page. Unfortunately the patch has still some problems , not latency-related: - The ISDN hisax driver (my card is a Fritz classic) crashes the kernel at modprobe hisax.o (does not happen on an unpatched 2.2.10 kernel) Can someone of the ISDN maintainers please reproduce/fix this ? (maybe a race at module initialization ?) - The disk performance decreases by 10-25% when I increase the CPU load in the "latencytest" bench. (On light CPU load there are no disk performance differences, maybe this is related to higher scheduling overhead) I think most of us want to have these "low-latency" features in the upcoming 2.4 kernel since it will make Linux a very good _MULTIMEDIA_OS_. With Mingo's patches the Linux low-latency performance comes very close to BEOS, and is much much better (3-4 times) Windows on the same hardware. It's now time to stress audio-software vendors to port their cool apps to Linux. comments ? PS: To Microsoft's Anti-Linux team: just download the patch and compare the performance with your crappy DirectX API :-) regards, Benno. -- Benno Senoner E-Mail: sbe...@gardena.net Linux scheduling latency benchmarks http://www.gardena.net/benno/linux/audio - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.rutgers.edu Please read the FAQ at http://www.tux.org/lkml/
From: yodai...@fsmlabs.com Subject: Re: [rtl] Low-latency patches working GREAT (<2.9ms audio latency), see testresults ,but ISDN troubles Date: 1999/08/28 Message-ID: <fa.f85ke2v.1f24tb8@ifi.uio.no>#1/1 X-Deja-AN: 518348561 Original-Date: Sat, 28 Aug 1999 17:55:10 -0600 Sender: owner-linux-ker...@vger.rutgers.edu Original-Message-ID: <19990828175510.A4356@hq.fsmlabs.com> References: <fa.k21ohcv.fm4h8u@ifi.uio.no> To: Benno Senoner <sbe...@gardena.net> Original-References: <99082823144300.00...@linuxhost.localdomain> Content-Type: text/plain; charset=us-ascii X-Orcpt: rfc822;linux-kernel-outgoing-dig Organization: Internet mailing list Mime-Version: 1.0 Newsgroups: fa.linux.kernel X-Loop: majord...@vger.rutgers.edu On Sat, Aug 28, 1999 at 10:40:57PM +0200, Benno Senoner wrote: > - The disk performance decreases by 10-25% when I increase the CPU load in > the "latencytest" bench. > (On light CPU load there are no disk performance differences, > maybe this is related to higher scheduling overhead) > > I think most of us want to have these "low-latency" features in the upcoming > 2.4 kernel since it will make Linux a very good _MULTIMEDIA_OS_. A 25% disk i/o decrease is very serious. Lets get some serious feedback from people running internet and database servers before we blow off the server users in order to compete with BEOS. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.rutgers.edu Please read the FAQ at http://www.tux.org/lkml/
From: Linus Torvalds <torva...@transmeta.com> Subject: Re: [rtl] Low-latency patches working GREAT (<2.9ms audio latency), see testresults ,but ISDN troubles Date: 1999/08/29 Message-ID: <fa.o9a3g7v.hkg6ra@ifi.uio.no>#1/1 X-Deja-AN: 518422414 Original-Date: Sat, 28 Aug 1999 23:21:42 -0700 (PDT) Sender: owner-linux-ker...@vger.rutgers.edu Original-Message-ID: <Pine.LNX.4.10.9908282320260.2144-100000@penguin.transmeta.com> References: <fa.f85ke2v.1f24tb8@ifi.uio.no> To: yodai...@fsmlabs.com X-Authentication-Warning: penguin.transmeta.com: torvalds owned process doing -bs Content-Type: TEXT/PLAIN; charset=US-ASCII X-Orcpt: rfc822;linux-kernel-outgoing-dig Organization: Internet mailing list MIME-Version: 1.0 Newsgroups: fa.linux.kernel X-Loop: majord...@vger.rutgers.edu On Sat, 28 Aug 1999 yodai...@fsmlabs.com wrote: > > A 25% disk i/o decrease is very serious. Lets get some serious feedback > from people running internet and database servers before we blow off > the server users in order to compete with BEOS. Guys, if anybody thinks we're competing with BeOS, then wake up. BeOS is a niche OS that isn't worth competing against, and at most we can try to find out what it's good at and see if we can emulate some of it. But 25% disk IO decrease is definitely not something we want to even consider. Linus - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.rutgers.edu Please read the FAQ at http://www.tux.org/lkml/
From: Ingo Molnar <mi...@chiara.csoma.elte.hu> Subject: Re: [rtl] Low-latency patches working GREAT (<2.9ms audio latency), see testresults ,but ISDN troubles Date: 1999/08/29 Message-ID: <fa.lgoji0v.f5s0g6@ifi.uio.no>#1/1 X-Deja-AN: 518430209 Original-Date: Sun, 29 Aug 1999 09:17:31 +0200 (CEST) Sender: owner-linux-ker...@vger.rutgers.edu Original-Message-ID: <Pine.LNX.3.96.990829091531.4857C-100000@chiara.csoma.elte.hu> References: <fa.gq5evnv.1p5un9c@ifi.uio.no> To: Alan Cox <a...@lxorguk.ukuu.org.uk> Content-Type: TEXT/PLAIN; charset=US-ASCII X-Orcpt: rfc822;linux-kernel-outgoing-dig Organization: Internet mailing list MIME-Version: 1.0 Newsgroups: fa.linux.kernel X-Loop: majord...@vger.rutgers.edu On Sun, 29 Aug 1999, Alan Cox wrote: > We also have tons of time. Its not a 2.2. candidate yes. between 2.2 and 2.3 there were so many fundamental changes that i will not even try to clean the 2.2 patch up - it would be too fundamental anyway. If anyone wants a solution for 2.2, the current patch can be used (it's 100% stable), but i'm developing the 'clean patch' only for 2.3. -- mingo - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.rutgers.edu Please read the FAQ at http://www.tux.org/lkml/
From: Ingo Molnar <mi...@chiara.csoma.elte.hu> Subject: Re: [rtl] Low-latency patches working GREAT (<2.9ms audio latency), see testresults ,but ISDN troubles Date: 1999/08/29 Message-ID: <fa.lgonj0v.f5k1g9@ifi.uio.no>#1/1 X-Deja-AN: 518432634 Original-Date: Sun, 29 Aug 1999 09:13:26 +0200 (CEST) Sender: owner-linux-ker...@vger.rutgers.edu Original-Message-ID: <Pine.LNX.3.96.990829085834.4857A-100000@chiara.csoma.elte.hu> References: <fa.o9a3g7v.hkg6ra@ifi.uio.no> To: Linus Torvalds <torva...@transmeta.com> Content-Type: TEXT/PLAIN; charset=US-ASCII X-Orcpt: rfc822;linux-kernel-outgoing-dig Organization: Internet mailing list MIME-Version: 1.0 Newsgroups: fa.linux.kernel X-Loop: majord...@vger.rutgers.edu On Sat, 28 Aug 1999, Linus Torvalds wrote: > Guys, if anybody thinks we're competing with BeOS, then wake up. BeOS is > a niche OS that isn't worth competing against, and at most we can try to > find out what it's good at and see if we can emulate some of it. But 25% > disk IO decrease is definitely not something we want to even consider. definitely. The patches are 'work in progress', and i know they are only acceptable for inclusion if they: 1) cause no measureable slowdown _anywhere_ 2) fixes 'buggy' latencies by redesigning the latency source, not by intruding the latency core with conditional reschedule points. i'm quite certain both 1) and 2) are very much possible, Benno's reported 25% slowdown (which btw. i think is not a pure bandwith degradation, but a slowdown in certain, not necesserily well understood circumstances) is simply a bug in my patch. the patch in it's current form is just a 'demo' that shows that good in-kernel latencies (the example to follow is not BeOS but QNX i think) are indeed possible without architectural impact. On my box i've killed all latencies that are bigger than 0.5 msec. (i'll try to fix the 2.9msec peak reported as well) I'll send an updated and fixed patch (in smaller pieces) for 2.3. -- mingo - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.rutgers.edu Please read the FAQ at http://www.tux.org/lkml/
From: Ingo Molnar <mi...@chiara.csoma.elte.hu> Subject: Re: Low-latency patches working GREAT (<2.9ms audio latency), see testresults ,but ISDN troubles Date: 1999/08/29 Message-ID: <fa.lfovi0v.e5g0gc@ifi.uio.no>#1/1 X-Deja-AN: 518432635 Original-Date: Sun, 29 Aug 1999 09:22:24 +0200 (CEST) Sender: owner-linux-ker...@vger.rutgers.edu Original-Message-ID: <Pine.LNX.3.96.990829091817.4857D-100000@chiara.csoma.elte.hu> References: <fa.k21ohcv.fm4h8u@ifi.uio.no> To: Benno Senoner <sbe...@gardena.net> Content-Type: TEXT/PLAIN; charset=US-ASCII X-Orcpt: rfc822;linux-kernel-outgoing-dig Organization: Internet mailing list MIME-Version: 1.0 Newsgroups: fa.linux.kernel X-Loop: majord...@vger.rutgers.edu On Sat, 28 Aug 1999, Benno Senoner wrote: > - The disk performance decreases by 10-25% when I increase the CPU load > in the "latencytest" bench. (On light CPU load there are no disk > performance differences, maybe this is related to higher scheduling > overhead) just in case anyone misunderstood the above result (as i think many did). The 'CPU load' mentioned above is a _realtime process_ (unless Benno you changed the benchmark). This is a _good_ result. 'increase CPU load' simply means 'generate more RT load' - which in _this case_ might mean less disk performance - but the RT task asked for it in the first place. All in one: i can see no problem here. if a 'simple' CPU-user (non-RT) is getting more active there is no slowdown. -- mingo - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.rutgers.edu Please read the FAQ at http://www.tux.org/lkml/
From: Paul Barton-Davis <p...@Op.Net> Subject: Re: [alsa-devel] Re: [rtl] Low-latency patches working GREAT (<2.9ms audio latency), see testresults ,but ISDN troubles Date: 1999/08/29 Message-ID: <fa.ehkmisv.1qmu0hl@ifi.uio.no>#1/1 X-Deja-AN: 518432636 Original-Date: Sun, 29 Aug 1999 03:34:41 -0400 Sender: owner-linux-ker...@vger.rutgers.edu Original-Message-Id: <199908290728.DAA29729@renoir.op.net> References: <fa.o9a3g7v.hkg6ra@ifi.uio.no> To: linux-ker...@vger.rutgers.edu, linux-so...@vger.rutgers.edu, alsa-de...@alsa-project.org X-Orcpt: rfc822;linux-kernel-outgoing-dig Organization: Internet mailing list Newsgroups: fa.linux.kernel X-Loop: majord...@vger.rutgers.edu Linus wrote: >Guys, if anybody thinks we're competing with BeOS, then wake up. BeOS is a last i heard, we weren't "competing" with anyone. we were trying to write a really really good OS. BeOS is unique in that its the first vaguely stable OS to be designed with multimedia in mind from the ground up. there are those of us who see computing technology as either currently or very soon delivering the power to do things with audio and video that have been hard to do without special purpose hardware as little as two years ago. right now, linux is not the right platform for doing this, and BeOS is, and I don't like that. there is persistent denial on l-k and elsewhere that an OS designed to effectively support "traditional" computer usage (compilation, number crunching, data serving) has any fundamental differences from one based around, say, real-time audio generation & manipulation. i doubt that this is true, but burying our heads in the sand won't change the answer one way or another. i personally suspect that the only reason BeOS has a niche *at all* is (1) because its true that there is a difference and (2) because there are companies out there who would like to develop for something better than Win/Mac and are scared to do the linux^H^H^H^H^Hright thing. it would be very nice to be able to compile linux with options that say, in essence: "this is a mostly single-user box with little or no network use, not much disk I/O that really benefits from a smart filesystem, and regular use of soft-real-time tasks that want to take over the machine if at all possible." this is a profile that has a very different emphasis from many of the things that kernel folks are working on supporting. to dismiss this kind of system as a "niche" is to potentially miss a broad wave in computing usage over the next decade or so. IMHO. >niche OS that isn't worth competing against, and at most we can try to >find out what it's good at and see if we can emulate some of it. But 25% "emulate" ? i'd rather we copy it and/or do it better. >disk IO decrease is definitely not something we want to even consider. oh come on linus - nobody has gotten close to suggesting that its an acceptable change! benno was merely observing the current status of things. i strongly suspect that victor's "tunable parameter" suggestion could solve quite a bit of the difference. --p - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.rutgers.edu Please read the FAQ at http://www.tux.org/lkml/
From: Benno Senoner <sbe...@gardena.net> Subject: Re: Low-latency patches working GREAT (<2.9ms audio latency), see testresults ,but ISDN troubles Date: 1999/08/29 Message-ID: <fa.k4hke4v.d68igj@ifi.uio.no>#1/1 X-Deja-AN: 518640661 Original-Date: Mon, 30 Aug 1999 01:41:58 +0200 Sender: owner-linux-ker...@vger.rutgers.edu Content-Transfer-Encoding: 7BIT Original-Message-Id: <99083001561703.00812@linuxhost.localdomain> References: <fa.lfovi0v.e5g0gc@ifi.uio.no> To: Ingo Molnar <mi...@chiara.csoma.elte.hu> Original-References: <Pine.LNX.3.96.990829091817.4857D-100...@chiara.csoma.elte.hu> Content-Type: text/plain; charset=US-ASCII X-Orcpt: rfc822;linux-kernel-outgoing-dig Organization: Internet mailing list MIME-Version: 1.0 Newsgroups: fa.linux.kernel X-Loop: majord...@vger.rutgers.edu On Sun, 29 Aug 1999, Ingo Molnar wrote: > On Sat, 28 Aug 1999, Benno Senoner wrote: > > > - The disk performance decreases by 10-25% when I increase the CPU load > > in the "latencytest" bench. (On light CPU load there are no disk > > performance differences, maybe this is related to higher scheduling > > overhead) > > just in case anyone misunderstood the above result (as i think many did). > The 'CPU load' mentioned above is a _realtime process_ (unless Benno you > changed the benchmark). This is a _good_ result. 'increase CPU load' > simply means 'generate more RT load' - which in _this case_ might mean > less disk performance - but the RT task asked for it in the first place. > All in one: i can see no problem here. Ingo, you are right: I just ran the unmodified latencytest and was measuring disk performace. In my example I used 1.45ms fragments, this means that my realtime thread has to be rescheduled about 600times/sec , and during each 1.45ms cycle the rt thread wastes about 1.1ms (= 80% of 1.45ms) of the CPU. If you lower the CPU load to low values ( 10% for example) , there is not a big substantial disk WRITE performance loss compared to an unpatched kernel. The interesting thing is that the disk READ performance is unaffected by your patches, on CPU loads of 80% I get the same disk READ performance on your lowlatency-N6 kernel as on a standard kernel. Perhaps the problem lies in your modifications in the disk write code ? > > if a 'simple' CPU-user (non-RT) is getting more active there is no > slowdown. I don't tested this yes, but I think you are right. (Will report back some numbers when I get the benchmarking done). You said your patch was 100% stable: are we talking about the same patches ? The latest I got was -N6 + shm.c backed out + Roger's conditional_schedule() wrapping in filemap.c. The kernel is rock solid , except it crashes when I do a modprobe of the ISDN hisax.o module. regards, Benno. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.rutgers.edu Please read the FAQ at http://www.tux.org/lkml/