Request for Testing: TCP RACK

[t***@freebsd.org]

Dear all,

recently the main branch was changed to build the TCP RACK stack
which is a loadable kernel module, by default:
https://cgit.FreeBSD.org/src/commit/?id=3a338c534154164504005beb00a3c6feb03756cc

As discussed on the bi-weekly transport call, it would be great if people
could test the RACK stack for their workload. Please report any problems to the
net@ mailing list or open an issue in the bug tracker and drop me a note via email.
This includes regressions in CPU usage, regressions in performance or any other
unexpected change you observe.

You can load the kernel module using
kldload tcp_rack

You can make the RACK stack the default stack using
sysctl net.inet.tcp.functions_default=rack

Based on the feedback we get, the default stack might be switched to the
RACK stack.

Please let me know if you have any questions.

Best regards
Michael




--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Herbert J. Skuhra]

Hi,

Post by t***@freebsd.org
Dear all,
recently the main branch was changed to build the TCP RACK stack
https://cgit.FreeBSD.org/src/commit/?id=3a338c534154164504005beb00a3c6feb03756cc
As discussed on the bi-weekly transport call, it would be great if people
could test the RACK stack for their workload. Please report any problems to the
This includes regressions in CPU usage, regressions in performance or any other
unexpected change you observe.
You can load the kernel module using
kldload tcp_rack
You can make the RACK stack the default stack using
sysctl net.inet.tcp.functions_default=rack
Based on the feedback we get, the default stack might be switched to the
RACK stack.
Please let me know if you have any questions.

I am running main-n266450-a592812327de with a GENERIC-NODEBUG kernel.
# kldload tcp_rack
kldload: an error occurred while loading module tcp_rack. Please check
dmesg(8) for more details.
KLD tcp_rack.ko: depends on tcphpts - not available or version mismatch
linker_load_file: /boot/kernel/tcp_rack.ko - unsupported file type
So you have to build a kernel with "options TCPHPTS" first?

OK, I am now running GENERIC-NODEBUG + "options TCPHPTS".

After setting "sysctl net.inet.tcp.functions_default=rack" git no
longer works:

# sysctl net.inet.tcp.functions_default=rack
$ git pull
client_loop: send disconnect: Broken pipe
fatal: the remote end hung up upon initial contact
# sysctl net.inet.tcp.functions_default=freebsd
$ git pull
Already up to date.

--
Herbert


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Olivier Cochard-Labbé]

Post by Herbert J. Skuhra
OK, I am now running GENERIC-NODEBUG + "options TCPHPTS".
After setting "sysctl net.inet.tcp.functions_default=rack" git no

Are you using a fresh 15 head or a specific network setup ?

Because I'm not able to reproduce your problem on my system:

$ uname -a
FreeBSD bigone 15.0-CURRENT FreeBSD 15.0-CURRENT #0
main-n266452-070d9e3540e6: Thu Nov 16 17:53:15 CET 2023
***@bigone:/usr/obj/usr/src/amd64.amd64/sys/TCPHPTS
amd64
$ cat /usr/src/sys/amd64/conf/TCPHPTS
include GENERIC-NODEBUG
ident TCPHPTS
options TCPHPTS
$ sysctl net.inet.tcp.functions_default
net.inet.tcp.functions_default: rack
$ git clone -q ***@github.com:freebsd/freebsd-src.git && echo working
working
$

[Herbert J. Skuhra]

Hi,

Post by Olivier Cochard-Labbé

Post by Herbert J. Skuhra
OK, I am now running GENERIC-NODEBUG + "options TCPHPTS".
After setting "sysctl net.inet.tcp.functions_default=rack" git no

Are you using a fresh 15 head or a specific network setup ?
$ uname -a
FreeBSD bigone 15.0-CURRENT FreeBSD 15.0-CURRENT #0
main-n266452-070d9e3540e6: Thu Nov 16 17:53:15 CET 2023
amd64
$ cat /usr/src/sys/amd64/conf/TCPHPTS
include GENERIC-NODEBUG
ident TCPHPTS
options TCPHPTS
$ sysctl net.inet.tcp.functions_default
net.inet.tcp.functions_default: rack
working
$

OK, (g)it works if I disable pf. Do you use pf?

Can you share your pf config such that I can reproduce the problem locally?

pass in all
pass out all
2. Fetching port distfiles also failed.
3. If I disable rxcsum on the ethernet adapter (igb0) it works.

Disabling lro also resolves the issue.

If I run "sysctl net.inet.tcp.rack.features.cmpack=0" I don't have to
disable rxcsum/tcxsum or lro on igb0.

--
Herbert


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[t***@freebsd.org]

Post by Herbert J. Skuhra

Hi,

Post by Olivier Cochard-Labbé

Post by Herbert J. Skuhra
OK, I am now running GENERIC-NODEBUG + "options TCPHPTS".
After setting "sysctl net.inet.tcp.functions_default=rack" git no

Are you using a fresh 15 head or a specific network setup ?
$ uname -a
FreeBSD bigone 15.0-CURRENT FreeBSD 15.0-CURRENT #0
main-n266452-070d9e3540e6: Thu Nov 16 17:53:15 CET 2023
amd64
$ cat /usr/src/sys/amd64/conf/TCPHPTS
include GENERIC-NODEBUG
ident TCPHPTS
options TCPHPTS
$ sysctl net.inet.tcp.functions_default
net.inet.tcp.functions_default: rack
working
$

OK, (g)it works if I disable pf. Do you use pf?

Can you share your pf config such that I can reproduce the problem locally?

pass in all
pass out all
2. Fetching port distfiles also failed.
3. If I disable rxcsum on the ethernet adapter (igb0) it works.

Disabling lro also resolves the issue.

If I run "sysctl net.inet.tcp.rack.features.cmpack=0" I don't have to
disable rxcsum/tcxsum or lro on igb0.

Does the problem also goes away if you disable pf completely, but keep
compressed acks enabled?

Best regards
Michael

Post by Herbert J. Skuhra
--
Herbert

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Herbert J. Skuhra]

Post by t***@freebsd.org

Post by Herbert J. Skuhra

Hi,

Post by Olivier Cochard-Labbé

Post by Herbert J. Skuhra
OK, I am now running GENERIC-NODEBUG + "options TCPHPTS".
After setting "sysctl net.inet.tcp.functions_default=rack" git no

Are you using a fresh 15 head or a specific network setup ?
$ uname -a
FreeBSD bigone 15.0-CURRENT FreeBSD 15.0-CURRENT #0
main-n266452-070d9e3540e6: Thu Nov 16 17:53:15 CET 2023
amd64
$ cat /usr/src/sys/amd64/conf/TCPHPTS
include GENERIC-NODEBUG
ident TCPHPTS
options TCPHPTS
$ sysctl net.inet.tcp.functions_default
net.inet.tcp.functions_default: rack
working
$

OK, (g)it works if I disable pf. Do you use pf?

Can you share your pf config such that I can reproduce the problem locally?

pass in all
pass out all
2. Fetching port distfiles also failed.
3. If I disable rxcsum on the ethernet adapter (igb0) it works.

Disabling lro also resolves the issue.

If I run "sysctl net.inet.tcp.rack.features.cmpack=0" I don't have to
disable rxcsum/tcxsum or lro on igb0.

Does the problem also goes away if you disable pf completely, but keep
compressed acks enabled?

Yes, it works with pf disabled and compressed acks enabled.

--
Herbert


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[void]

Post by t***@freebsd.org
You can load the kernel module using
kldload tcp_rack
You can make the RACK stack the default stack using
sysctl net.inet.tcp.functions_default=rack

Hi, thank you for this.

https://klarasystems.com/articles/using-the-freebsd-rack-tcp-stack/ mentions
this needs to be set in /etc/src.conf :

WITH_EXTRA_TCP_STACKS=1

Is this still the case? Context here is -current both in a vm and bare
metal, on various machines, on various connections, from DSL to 10Gb.

Is there a method (yet) for enabling this functionality in various -RELENG
maybe where one can compile in a vm built for that purpose, then
transferring to the production vm?

Would it be expected to work on arm64?

What testing would you like doing?

[Alexander Leidinger]

Post by void

Post by t***@freebsd.org
You can load the kernel module using
kldload tcp_rack
You can make the RACK stack the default stack using
sysctl net.inet.tcp.functions_default=rack

Hi, thank you for this.
https://klarasystems.com/articles/using-the-freebsd-rack-tcp-stack/ mentions
WITH_EXTRA_TCP_STACKS=1
Is this still the case? Context here is -current both in a vm and bare
metal, on various machines, on various connections, from DSL to 10Gb.

On a recent -current: this is not needed anymore, it is part of the
defaults now. But you may still compile the kernel with "option TCPHPTS"
(until it's added to the defaults too).

Post by void
Is there a method (yet) for enabling this functionality in various -RELENG
maybe where one can compile in a vm built for that purpose, then
transferring to the production vm?

Copy the kernel which was build according to the acticle from klara
systems to your target VM.

Post by void
Would it be expected to work on arm64?

Yes (I use it on an ampere VM in the cloud).

Bye,
Alexander.

[t***@freebsd.org]

Post by Tomoaki AOKI
On Fri, 17 Nov 2023 18:51:05 +0100

Post by t***@freebsd.org

I am running the rack stack for quiet some time now on a baremetal machiene and never had problems.
Also use pf. This is a test machine so not a lot happening on it.
Are there any thing we can test? Do we have some test scripts we can run?

We are actually interested in feedback about using the stack in whatever
use case you use TCP for. The stack has been tested with the Netflix use
case, but not so much with others. That is why we ask for broader testing.
Best regards
Michael

Are there any difference regarding with performance between main and
stable/14? If so, please ignore below.
I have stable/14 environment which is configured to be able to switch
to TCP RACK and experienced huge performance loss when writing
a large file to smbfs share on commercial NAS. CC is cubic.
Testing large archive on the smbfs share doesn't seem to be
affected.
Comparison between default (freebsd) and rack TCP stack using
sysutils/clone on stable/14 at commit 7d1321288ad9, amd64.
Last 3 lines of outputs from clone (upload to NAS) are shown.

Thank you very much for testing. This is what we are looking
for. Would it be possible to repeat the test using NewReno as
the CC?

Best regards
Michael

Post by Tomoaki AOKI
1 item copied, 2342.4 MB in 39.12 s -- 59.9 MB/s
Leaked memory: 0 bytes
No errors occured.
1 item copied, 2342.4 MB in 926.59 s -- 2.5 MB/s
Leaked memory: 0 bytes
No errors occured.
1 item copied, 2342.4 MB in 906.94 s -- 2.6 MB/s
Leaked memory: 0 bytes
No errors occured.
1 item copied, 2342.4 MB in 39.12 s -- 59.9 MB/s
Leaked memory: 0 bytes
No errors occured.
Regards.
--

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Tomoaki AOKI]

On Sat, 18 Nov 2023 09:50:43 +0100

Post by t***@freebsd.org

Post by Tomoaki AOKI
On Fri, 17 Nov 2023 18:51:05 +0100

Post by t***@freebsd.org

I am running the rack stack for quiet some time now on a baremetal machiene and never had problems.
Also use pf. This is a test machine so not a lot happening on it.
Are there any thing we can test? Do we have some test scripts we can run?

We are actually interested in feedback about using the stack in whatever
use case you use TCP for. The stack has been tested with the Netflix use
case, but not so much with others. That is why we ask for broader testing.
Best regards
Michael

Are there any difference regarding with performance between main and
stable/14? If so, please ignore below.
I have stable/14 environment which is configured to be able to switch
to TCP RACK and experienced huge performance loss when writing
a large file to smbfs share on commercial NAS. CC is cubic.
Testing large archive on the smbfs share doesn't seem to be
affected.
Comparison between default (freebsd) and rack TCP stack using
sysutils/clone on stable/14 at commit 7d1321288ad9, amd64.
Last 3 lines of outputs from clone (upload to NAS) are shown.

Thank you very much for testing. This is what we are looking
for. Would it be possible to repeat the test using NewReno as
the CC?
Best regards
Michael

Sure. Here we go!

sysctl net.inet.tcp.functions_default
net.inet.tcp.functions_default: freebsd
sysctl net.inet.tcp.cc.algorithm
net.inet.tcp.cc.algorithm: newreno
Umounted and remounted smbfs share.

1 item copied, 2343.1 MB in 37.65 s -- 62.2 MB/s
Leaked memory: 0 bytes
No errors occured.


sysctl net.inet.tcp.functions_default
net.inet.tcp.functions_default: rack
Umounted and remounted smbfs share.

1 item copied, 2343.1 MB in 905.17 s -- 2.6 MB/s
Leaked memory: 0 bytes
No errors occured.


sysctl net.inet.tcp.functions_default
net.inet.tcp.functions_default: freebsd
Without umount and remount to reproduce previous oddness, maybe caused
by keep-alive.

1 item copied, 2343.1 MB in 897.67 s -- 2.6 MB/s
Leaked memory: 0 bytes
No errors occured.


Umounted and remounted, without change for CC and TCP stack.

1 item copied, 2343.1 MB in 37.43 s -- 62.6 MB/s
Leaked memory: 0 bytes
No errors occured.


All test are proceeded simultaneously. So the last one is with
CC=newreno and TCP stack=freebsd.

Not exactly recorded, but testing transferred file by
diff -u -p -N was roughly 30MB/sec, while roughly 25MB/sec with
CC=cubic.

Post by t***@freebsd.org

Post by Tomoaki AOKI
1 item copied, 2342.4 MB in 39.12 s -- 59.9 MB/s
Leaked memory: 0 bytes
No errors occured.
1 item copied, 2342.4 MB in 926.59 s -- 2.5 MB/s
Leaked memory: 0 bytes
No errors occured.
1 item copied, 2342.4 MB in 906.94 s -- 2.6 MB/s
Leaked memory: 0 bytes
No errors occured.
1 item copied, 2342.4 MB in 39.12 s -- 59.9 MB/s
Leaked memory: 0 bytes
No errors occured.
Regards.
--

[t***@freebsd.org]

I am running the rack stack for quiet some time now on a baremetal machiene and never had problems.
Also use pf. This is a test machine so not a lot happening on it.
Are there any thing we can test? Do we have some test scripts we can run?

We are actually interested in feedback about using the stack in whatever
use case you use TCP for. The stack has been tested with the Netflix use
case, but not so much with others. That is why we ask for broader testing.

Best regards
Michael
--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Tomoaki AOKI]

On Fri, 17 Nov 2023 18:51:05 +0100

Post by t***@freebsd.org

I am running the rack stack for quiet some time now on a baremetal machiene and never had problems.
Also use pf. This is a test machine so not a lot happening on it.
Are there any thing we can test? Do we have some test scripts we can run?

We are actually interested in feedback about using the stack in whatever
use case you use TCP for. The stack has been tested with the Netflix use
case, but not so much with others. That is why we ask for broader testing.
Best regards
Michael

Are there any difference regarding with performance between main and
stable/14? If so, please ignore below.

I have stable/14 environment which is configured to be able to switch
to TCP RACK and experienced huge performance loss when writing
a large file to smbfs share on commercial NAS. CC is cubic.
Testing large archive on the smbfs share doesn't seem to be
affected.

Comparison between default (freebsd) and rack TCP stack using
sysutils/clone on stable/14 at commit 7d1321288ad9, amd64.
Last 3 lines of outputs from clone (upload to NAS) are shown.

Before switching to rack:
1 item copied, 2342.4 MB in 39.12 s -- 59.9 MB/s
Leaked memory: 0 bytes
No errors occured.

Unmount the smbfs share, switch to rack, and after remount:
1 item copied, 2342.4 MB in 926.59 s -- 2.5 MB/s
Leaked memory: 0 bytes
No errors occured.

Switch back to freebsd (default) without unmounting:
1 item copied, 2342.4 MB in 906.94 s -- 2.6 MB/s
Leaked memory: 0 bytes
No errors occured.

Unmount and remount the smbfs share:
1 item copied, 2342.4 MB in 39.12 s -- 59.9 MB/s
Leaked memory: 0 bytes
No errors occured.


Regards.

[Tomoaki AOKI]

On Sun, 19 Nov 2023 04:01:01 +0900

Post by Tomoaki AOKI
On Sat, 18 Nov 2023 09:50:43 +0100

Post by t***@freebsd.org

Post by Tomoaki AOKI
On Fri, 17 Nov 2023 18:51:05 +0100

Post by t***@freebsd.org

I am running the rack stack for quiet some time now on a baremetal machiene and never had problems.
Also use pf. This is a test machine so not a lot happening on it.
Are there any thing we can test? Do we have some test scripts we can run?

We are actually interested in feedback about using the stack in whatever
use case you use TCP for. The stack has been tested with the Netflix use
case, but not so much with others. That is why we ask for broader testing.
Best regards
Michael

Are there any difference regarding with performance between main and
stable/14? If so, please ignore below.
I have stable/14 environment which is configured to be able to switch
to TCP RACK and experienced huge performance loss when writing
a large file to smbfs share on commercial NAS. CC is cubic.
Testing large archive on the smbfs share doesn't seem to be
affected.
Comparison between default (freebsd) and rack TCP stack using
sysutils/clone on stable/14 at commit 7d1321288ad9, amd64.
Last 3 lines of outputs from clone (upload to NAS) are shown.

Thank you very much for testing. This is what we are looking
for. Would it be possible to repeat the test using NewReno as
the CC?
Best regards
Michael

Sure. Here we go!
sysctl net.inet.tcp.functions_default
net.inet.tcp.functions_default: freebsd
sysctl net.inet.tcp.cc.algorithm
net.inet.tcp.cc.algorithm: newreno
Umounted and remounted smbfs share.
1 item copied, 2343.1 MB in 37.65 s -- 62.2 MB/s
Leaked memory: 0 bytes
No errors occured.
sysctl net.inet.tcp.functions_default
net.inet.tcp.functions_default: rack
Umounted and remounted smbfs share.
1 item copied, 2343.1 MB in 905.17 s -- 2.6 MB/s
Leaked memory: 0 bytes
No errors occured.
sysctl net.inet.tcp.functions_default
net.inet.tcp.functions_default: freebsd
Without umount and remount to reproduce previous oddness, maybe caused
by keep-alive.
1 item copied, 2343.1 MB in 897.67 s -- 2.6 MB/s
Leaked memory: 0 bytes
No errors occured.
Umounted and remounted, without change for CC and TCP stack.
1 item copied, 2343.1 MB in 37.43 s -- 62.6 MB/s
Leaked memory: 0 bytes
No errors occured.
All test are proceeded simultaneously. So the last one is with
CC=newreno and TCP stack=freebsd.
Not exactly recorded, but testing transferred file by
diff -u -p -N was roughly 30MB/sec, while roughly 25MB/sec with
CC=cubic.

Post by t***@freebsd.org

Post by Tomoaki AOKI
1 item copied, 2342.4 MB in 39.12 s -- 59.9 MB/s
Leaked memory: 0 bytes
No errors occured.
1 item copied, 2342.4 MB in 926.59 s -- 2.5 MB/s
Leaked memory: 0 bytes
No errors occured.
1 item copied, 2342.4 MB in 906.94 s -- 2.6 MB/s
Leaked memory: 0 bytes
No errors occured.
1 item copied, 2342.4 MB in 39.12 s -- 59.9 MB/s
Leaked memory: 0 bytes
No errors occured.
Regards.
--

--

Just a follow up.

This situation does not changed on stable/14, amd64 until commit
a15b8e32942b2cbf70c7fc71e9c82d2b292e82c3. (So I've never reported again
until now. Not tested on every updates.)

tcphpts.ko is loaded.

Note that options RATELIMIT and options TCPHPTS are dropped when
changes to RACK is MFC'ed and added tcphpts_load="YES"
in /boot/loader.conf instaad.

Another note:
Differences between CC=newreno and CC=cubic on testing transferred file
seems to be just a fudge factor. both vaies mostly between the two
values which I've previously reported.

[Zhenlei Huang]

Post by t***@freebsd.org
Dear all,
recently the main branch was changed to build the TCP RACK stack
https://cgit.FreeBSD.org/src/commit/?id=3a338c534154164504005beb00a3c6feb03756cc
As discussed on the bi-weekly transport call, it would be great if people
could test the RACK stack for their workload. Please report any problems to the
This includes regressions in CPU usage, regressions in performance or any other
unexpected change you observe.

I see some performance regression with the rack stack.

This is iperf3 test on local host (bare metal, an old i5 2 cores 4 threads MBP).

freebsd: 16.1 Gbits/sec
rack: 12.3 Gbits/sec

The congestion control algorithm is cubic.

Note this is a quick test with DEBUG options enabled.

I'll try with no debug options and report later.

Post by t***@freebsd.org
You can load the kernel module using
kldload tcp_rack
You can make the RACK stack the default stack using
sysctl net.inet.tcp.functions_default=rack
Based on the feedback we get, the default stack might be switched to the
RACK stack.
Please let me know if you have any questions.
Best regards
Michael

Best regards,
Zhenlei



--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Zhenlei Huang]

Post by Zhenlei Huang

Post by t***@freebsd.org
Dear all,
recently the main branch was changed to build the TCP RACK stack
https://cgit.FreeBSD.org/src/commit/?id=3a338c534154164504005beb00a3c6feb03756cc
As discussed on the bi-weekly transport call, it would be great if people
could test the RACK stack for their workload. Please report any problems to the
This includes regressions in CPU usage, regressions in performance or any other
unexpected change you observe.

I see some performance regression with the rack stack.
This is iperf3 test on local host (bare metal, an old i5 2 cores 4 threads MBP).
freebsd: 16.1 Gbits/sec
rack: 12.3 Gbits/sec
The congestion control algorithm is cubic.
Note this is a quick test with DEBUG options enabled.
I'll try with no debug options and report later.

** UPDATE **

With no debug options:

freebsd: 37.2 Gbits/sec
rack: 27.9 Gbits/sec

Post by Zhenlei Huang

Post by t***@freebsd.org
You can load the kernel module using
kldload tcp_rack
You can make the RACK stack the default stack using
sysctl net.inet.tcp.functions_default=rack
Based on the feedback we get, the default stack might be switched to the
RACK stack.
Please let me know if you have any questions.
Best regards
Michael

Best regards,
Zhenlei

[Pete Wright]

Hello,
I'm curious about tcp RACK.
As I do not run on a server background, only a laptop and a rpi4 for
poudriere, git, browsing, some torrent and ssh/sftp connections, will
I see any difference using RACK?
What tests should I do for comparison?

I found this blog post from Klara was a good backgrounder to get my
comfortable with testing out RACK:
https://klarasystems.com/articles/using-the-freebsd-rack-tcp-stack/

I've been using it on a busy'ish server and a workstation without any
issues, but I'll defer to others as to what areas of focus for testing
are needed.

-pete

[t***@freebsd.org]

Hello,
I'm curious about tcp RACK.
As I do not run on a server background, only a laptop and a rpi4 for
poudriere, git, browsing, some torrent and ssh/sftp connections, will
I see any difference using RACK?
What tests should I do for comparison?

You might want to read the following article in the FreeBSD Journal:
https://freebsdfoundation.org/our-work/journal/browser-based-edition/rack-and-alternate-tcp-stacks-for-freebsd/

There is no specific area for testing. Just test the stack on
the systems you use with the workload you use and report back
any issues...

Best regards
Michael

Thanks,

Post by t***@freebsd.org
Dear all,
recently the main branch was changed to build the TCP RACK stack
https://cgit.FreeBSD.org/src/commit/?id=3a338c534154164504005beb00a3c6feb03756cc
As discussed on the bi-weekly transport call, it would be great if people
could test the RACK stack for their workload. Please report any problems to the
This includes regressions in CPU usage, regressions in performance or any other
unexpected change you observe.
You can load the kernel module using
kldload tcp_rack
You can make the RACK stack the default stack using
sysctl net.inet.tcp.functions_default=rack
Based on the feedback we get, the default stack might be switched to the
RACK stack.
Please let me know if you have any questions.
Best regards
Michael

--
Nuno Teixeira
FreeBSD Committer (ports)

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Nuno Teixeira]

Hello all,

On a laptop i7/16MB ram, desktop use and port testing (poudriere) I've
noticed that all operations on OS was increased by 3 to 5 times
longer.
examples:
- firefox took way long time to open
- poudriere testport tooked up to 3 times longer to finished

make.conf:
KERNCONF=GENERIC-NODEBUG
src.conf:
WITH_MALLOC_PRODUCTION=yes

tested on main-n268800-6a6ec90681cf

Thanks,

Post by Gary Jennejohn
In the article we have option TCPHPTS and option TCP_RACK. But in
/sys/conf/NOTES we have options TCPHPTS and options TCP_RACK and
not option.

Thanks for reporting, that is a typo in the article. It should
always read options instead of option.

It's not a typo, both spellings work, cf. config(5).

option in the article (for whatever reason) and tested the
configs using options...
Best regards
Michael

DES
--

[Nuno Teixeira]

Will update amd64 laptop to main-n268827-75464941dc17 (Mar 16) and test it.

Please do so...

@main-n268827-75464941dc17 GENERIC-NODEBUG amd64

Ok, I think I have here some numbers related to disk performance being
affected by tcp_rack that somehow is messing with something.

NOTES:
- test [1] was done after a boot without tcp_rack in loader.conf and
[2] tcp_rack was loaded manually with kldload (without rebooting)
- After unloading tcp_rack, same results as [2]
- Cannot unload tcptpts: device busy

[1] without tcp_rack loaded:

==> poudriere testport net/gitup:
11.16s real 5.35s user 6.35s sys

==> bonnie++:
Version 1.98 ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Name:Size etc /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
leg.home 32G 25k 99 105m 99 88.7m 99 77k 99 198m 99 12716 1784
Latency 351ms 1793us 2340us 241ms 638us 2514us
Version 1.98 ------Sequential Create------ --------Random Create--------
leg.home -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 14908.273230 97 +++++ +++ 6878.516657 99
15194.808063 97 +++++ +++ 8019.731670 99
Latency 1108us 182us 2228us 1013us 152us 2424us

[2] kldload tcp_rack:

==> poudriere testport net/gitup:
1m0.54s real 4.98s user 1m31.38s sys

==> bonnie++:
Version 1.98 ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Name:Size etc /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
leg.home 32G 14k 99 78.0m 99 46.6m 99 25k 99 120m 99 6688 2161
Latency 676ms 18309us 76171us 385ms 924us 2274us
Version 1.98 ------Sequential Create------ --------Random Create--------
leg.home -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 8139.513260 96 19437.792294 99 5494.638508 99
8099.275425 96 19723.528878 99 6363.123671 99
Latency 2982us 338us 3072us 1135us 591us 3236us

Cheers,







--
Nuno Teixeira
FreeBSD Committer (ports)


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[David Wolfskill]

...

It would be so nice that we can have a sysctl tunnable for this patch
so we could do more tests without recompiling kernel.

Thanks for testing!
@gallatin: can you come up with a patch that is acceptable for Netflix
and allows to mitigate the performance regression.

Ideally, tcphpts could enable this automatically when it starts to be
used (enough?), but a sysctl could select auto/on/off.
Mike
....

Or (at some risk of over-{complicat,engineer}ing things), perhaps
sysctl/tunable for high- and low-water marks?

Peace,
david (who is quite unlikely to be writing the code, so "grain of salt")

[Gary Jennejohn]

On Sat, 16 Mar 2024 09:49:24 +0000

Hello Gary,
Nice that you found this.
tcp_tack manual doesn't mention that we need extra config in kernel
but it loads module and it is shown in sysctl
net.inet.tcp.functions_available without errors.
I will add missing config to GENERIC and see how it goes.

All the required information is here:

https://freebsdfoundation.org/our-work/journal/browser-based-edition/rack-and-alternate-tcp-stacks-for-freebsd/

--
Gary Jennejohn


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Gary Jennejohn]

On Sat, 16 Mar 2024 10:11:22 +0000

(...)

makeoptions WITH_EXTRA_TCP_STACKS=1

From https://cgit.freebsd.org/src/commit/?id=3a338c534154164504005beb00a3c6feb03756cc
WITH_EXTRA_TCP_STACKS was dropped.

options TCPHPTS

That's the missing option in GENERIC that could lead to my slow
opearations problem

options TCP_RACK

Don't think I need this one as I will use kernel module instead of
building it in kernel.
options TCPHPTS
Is this correct?

Yeah, that will probably fix it. According to a comment in
/usr/src/sys/netinet/tcp_hpts.c it adds a high precision timer
system for tcp, used by RACK and BBR.

--
Gary Jennejohn


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Tomoaki AOKI]

On Sun, 17 Mar 2024 11:40:54 -0400

Resending with the patch as an attachment.
Drew

I don't have the full context, but it seems like the complaint is a performance regression in bonnie++ and perhaps other things when tcp_hpts is loaded, even when it is not used. Is that correct?
diff --git a/sys/kern/subr_trap.c b/sys/kern/subr_trap.c
index e9a16cd0b36e..54b540c97123 100644
--- a/sys/kern/subr_trap.c
+++ b/sys/kern/subr_trap.c
@@ -138,7 +138,7 @@ userret(struct thread *td, struct trapframe *frame)
* Software Timer Support for Network Processing"
* by Mohit Aron and Peter Druschel.
*/
- tcp_hpts_softclock();
+ /*tcp_hpts_softclock();*/
/*
* Let the scheduler adjust our priority etc.
*/
If that fixes it, I suspect we should either make this hook optional for casual users of tcp_hpts(), or add some kind of "last called" timestamp to prevent it being called over and over and over on workloads which are syscall heavy.
Note that for non-casual users of hpts (like Netflix, with hundreds of thousands of TCP connections managed by hpts), this call is a huge win, so I think we'd prefer that it remain in some form.
Drew

Controlled via RW or RWTUN sysctl/tunable?

[Nuno Teixeira]

Post by Gary Jennejohn

options TCPHPTS
Is this correct?

Yeah, that will probably fix it. According to a comment in
/usr/src/sys/netinet/tcp_hpts.c it adds a high precision timer
system for tcp, used by RACK and BBR.

As tuexen said, on main, loader.conf: tcp_rack_load="YES" will load
tcphpts.ko as I am seing in my rpi4 right now.
I'm testing it and check its performance.

I will test again on my amd64 laptop and run more tests too.

[Gary Jennejohn]

On Tue, 12 Mar 2024 20:14:17 +0100

Post by t***@freebsd.org

Hello,
I'm curious about tcp RACK.
As I do not run on a server background, only a laptop and a rpi4 for
poudriere, git, browsing, some torrent and ssh/sftp connections, will
I see any difference using RACK?
What tests should I do for comparison?

https://freebsdfoundation.org/our-work/journal/browser-based-edition/rack-and-alternate-tcp-stacks-for-freebsd/
There is no specific area for testing. Just test the stack on
the systems you use with the workload you use and report back
any issues...

In the article we have option TCPHPTS and option TCP_RACK. But in
/sys/conf/NOTES we have options TCPHPTS and options TCP_RACK and
not option.

--
Gary Jennejohn


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[t***@freebsd.org]

Post by Nuno Teixeira
Hello all,
On a laptop i7/16MB ram, desktop use and port testing (poudriere) I've
noticed that all operations on OS was increased by 3 to 5 times
longer.
- firefox took way long time to open
- poudriere testport tooked up to 3 times longer to finished

How did you enable the RACK stack? Does the poudriere involve
network interaction?

Best regards
Michael

Post by Nuno Teixeira
KERNCONF=GENERIC-NODEBUG
WITH_MALLOC_PRODUCTION=yes
tested on main-n268800-6a6ec90681cf
Thanks,

Post by Gary Jennejohn
In the article we have option TCPHPTS and option TCP_RACK. But in
/sys/conf/NOTES we have options TCPHPTS and options TCP_RACK and
not option.

Thanks for reporting, that is a typo in the article. It should
always read options instead of option.

It's not a typo, both spellings work, cf. config(5).

option in the article (for whatever reason) and tested the
configs using options...
Best regards
Michael

DES
--

--
Nuno Teixeira
FreeBSD Committer (ports)

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de