On Sun, Nov 19, 2017 at 03:09:38PM -0500, Eduard Nicodei wrote:
> >-------- Original Message --------
> >Subject: Re: sunxi/sxie.c: ~3KB/s TX speed (6.2 + uboot 17.03)
> >Local Time: November 14, 2017 12:07 AM
> >UTC Time: November 14, 2017 12:07 AM
> >From: artturi....@gmail.com
> >To: Eduard Nicodei <enico...@protonmail.com>
> >arm@openbsd.org
> >
> >On Mon, Nov 13, 2017 at 06:33:48PM -0500, Eduard Nicodei wrote:
> >>Hi,
> >>I've had same problem as Artturi
> >>(https://marc.info/?l=openbsd-bugs&m=150071170630651&w=2), but on a
> >>Olinuxino A10 Lime. After trying u-boot 17.03 I noticed that TX speeds
> >>were very slow (~3KB/s).
> >>Here is what the packets look like on the other machine (192.168.1.17):
> >>192.168.1.3.1484 > 192.168.1.17.30000: S 3400651399:3400651399(0) win 16384
> >> 192.168.1.17.30000 > 192.168.1.3.1484: S 2057093871:2057093871(0) ack
> >> 3400651400 win 28960
> >> 192.168.1.3.1484 > 192.168.1.17.30000: . ack 1 win 256
> >> 0b:90:d0:de:44:1c 5f:bd:8a:25:ff:c6 69ef 118:
> >> e506 fdff 29fe dfef 0094 3fff 0894 ffee
> >> a0a9 f9ab 8446 dcf3 ba20 5c47 0249 f3de
> >> c90a d7d1 0028 9ef4 9381 8b69 7103 8ef4
> >> 6e68 f7
> >> 192.168.1.3.1484 > 192.168.1.17.30000: . 1:1449(1448) ack 1 win 256
> >> 192.168.1.17.30000 > 192.168.1.3.1484: . ack 1449 win 249
> >> 0b:90:d0:de:44:1c 5f:bd:8a:25:ff:c6 69ef 1514:
> >> e506 fdff 29fe dfef 0094 3fff 0894 ffee
> >> a0a9 f9ab 8446 dcf3 ba20 5c47 0249 f3de
> >> c90a d7d1 0028 9ef4 9381 8b69 7103 8ef4
> >> 6e68 f7
> >> 192.168.1.3.1484 > 192.168.1.17.30000: . 1501:2949(1448) ack 1 win 256
> >> 192.168.1.17.30000 > 192.168.1.3.1484: . ack 1449 win 272
> >> 192.168.1.3.1484 > 192.168.1.17.30000: . 4397:5845(1448) ack 1 win 256
> >> 192.168.1.17.30000 > 192.168.1.3.1484: . ack 1449 win 295
> >>On the Olinuxino Lime (192.168.1.3) they look OK:
> >>192.168.1.3.1484 > 192.168.1.17.30000: S 3400651399:3400651399(0) win 16384
> >> 192.168.1.17.30000 > 192.168.1.3.1484: S 2057093871:2057093871(0) ack
> >> 3400651400 win 28960
> >> 192.168.1.3.1484 > 192.168.1.17.30000: . ack 1 win 256
> >> 192.168.1.3.1484 > 192.168.1.17.30000: . 1:1449(1448) ack 1 win 256
> >> 192.168.1.3.1484 > 192.168.1.17.30000: P 1449:1501(52) ack 1 win 256
> >> 192.168.1.3.1484 > 192.168.1.17.30000: . 1501:2949(1448) ack 1 win 256
> >> 192.168.1.3.1484 > 192.168.1.17.30000: . 2949:4397(1448) ack 1 win 256
> >> 192.168.1.3.1484 > 192.168.1.17.30000: . 4397:5845(1448) ack 1 win 256
> >> 192.168.1.3.1484 > 192.168.1.17.30000: . 5845:7293(1448) ack 1 win 256
> >> 192.168.1.17.30000 > 192.168.1.3.1484: . ack 1449 win 249
> >> 192.168.1.3.1484 > 192.168.1.17.30000: . 7293:8741(1448) ack 1 win 256
> >> 192.168.1.17.30000 > 192.168.1.3.1484: . ack 1449 win 272
> >> 192.168.1.3.1484 > 192.168.1.17.30000: . 8741:10189(1448) ack 1 win 256
> >> 192.168.1.17.30000 > 192.168.1.3.1484: . ack 1449 win 295
> >> 192.168.1.3.1484 > 192.168.1.17.30000: . 10189:11637(1448) ack 1 win 256
> >> 192.168.1.17.30000 > 192.168.1.3.1484: . ack 1449 win 317
> >>I believe this might be caused by sxie driver using two TX FIFOs: I hacked
> >>a bit the kernel and when using only 1 TX FIFO it did not send any mangled
> >>frames (though packets were getting dropped by ifq_enqueue). I also added
> >>some counters to keep track which queue was being used and the numbers of
> >>mangled frames were close the number of times the second FIFO was used.
> >>I would like to investigate further, but I would like to know if this could
> >>be caused by the older u-boot or if it seems like an actual fault with the
> >>driver?
> >>Has anybody had good TX speed using sxie? I saw John DiMarco also reported
> >>a TX speed of about 4KB/s but that was over a year ago
> >>(https://marc.info/?l=openbsd-arm&m=147378434716041&w=2).
> >>I booted a debian image and it run at a few *MB/s so I'm pretty confident
> >>it's not a hw problem.
> >>Many Thanks,
> >> Eduard
> >>
> >I can get ip and even ssh in on -current and latest u-boot,
> > but yeah it's broken enough that i did shutdown my boards w/it,
> > i know it used to do way more IRQs too.
> > I don't know yet, nor really feel like it'd be worth my time to do
> > anything than install something else on it, but it might be related
> > that someone did fuckup the timers on it too, tick does around 66/update
> > in systat, and stattick barely 100.
> >
> > fwiw., i got some fixes commited to u-boot for sun4i_emac,
> > but otherwise it's unable to boot sd0a:/bsd w/bootarm.efi,
> > so still broken&useless, it will be(U-Boot 2017.11).
> >
> > -Artturi
> >>U-Boot SPL 2017.03 (Apr 03 2017 - 11:14:00)
> >> DRAM: 512 MiB
> >> CPU: 912000000Hz, AXI/AHB/APB: 3/2/2
> >> Trying to boot from MMC1
> >>U-Boot 2017.03 (Apr 03 2017 - 11:14:00 -0600) Allwinner Technology
> >>CPU: Allwinner A10 (SUN4I)
> >> Model: Olimex A10-OLinuXino-LIME
> >> I2C: ready
> >> DRAM: 512 MiB
> >> MMC: SUNXI SD/MMC: 0
> >> *** Warning - bad CRC, using default environment
> >>In: serial
> >> Out: serial
> >> Err: serial
> >> SCSI: SATA link 0 timeout.
> >> AHCI 0001.0100 32 slots 1 ports 3 Gbps 0x1 impl SATA mode
> >> flags: ncq stag pm led clo only pmp pio slum part ccc apst
> >> Net: eth0: ethernet@01c0b000
> >> starting USB...
> >> USB0: USB EHCI 1.00
> >> USB1: USB OHCI 1.0
> >> USB2: USB EHCI 1.00
> >> USB3: USB OHCI 1.0
> >> scanning bus 0 for devices... 1 USB Device(s) found
> >> scanning bus 2 for devices... 1 USB Device(s) found
> >> scanning usb for storage devices... 0 Storage Device(s) found
> >> Hit any key to stop autoboot: 0
> >> switch to partitions #0, OK
> >> mmc0 is current device
> >> Scanning mmc 0:1...
> >> reading /sun4i-a10-olinuxino-lime.dtb
> >> 26581 bytes read in 30 ms (865.2 KiB/s)
> >> Found EFI removable media binary efi/boot/bootarm.efi
> >> reading efi/boot/bootarm.efi
> >> 67436 bytes read in 42 ms (1.5 MiB/s)
> >>Starting EFI application at 42000000 ...
> >>
> >>Scanning disks on scsi...
> >> Scanning disks on usb...
> >> Scanning disks on mmc...
> >> MMC Device 1 not found
> >> MMC Device 2 not found
> >> MMC Device 3 not found
> >> Found 6 disks
> >>>>OpenBSD/armv7 BOOTARM 1.0
> >>>> boot>
> >>>> booting sd0a:/bsd: 3935884+166328+562136
> >>>> [273417+90+522736+245831]=0x5785f4
> >>>>OpenBSD/armv7 booting ...
> >> arg0 0xc08785f4 arg1 0x0 arg2 0x48000000
> >> Allocating page tables
> >> freestart = 0x40879000, free_pages = 128903 (0x0001f787)
> >> IRQ stack: p0x408a7000 v0xc08a7000
> >> ABT stack: p0x408a8000 v0xc08a8000
> >> UND stack: p0x408a9000 v0xc08a9000
> >> SVC stack: p0x408aa000 v0xc08aa000
> >> Creating L1 page table at 0x4087c000
> >> Mapping kernel
> >> Constructing L2 page tables
> >> undefined page pmap [ using 1042532 bytes of bsd ELF symbol table ]
> >> board type: 0
> >> Copyright (c) 1982, 1986, 1989, 1991, 1993
> >> The Regents of the University of California. All rights reserved.
> >> Copyright (c) 1995-2017 OpenBSD. All rights reserved.
> >> https://www.OpenBSD.org
> >>OpenBSD 6.2-current (GENERIC) #122: Sat Nov 11 11:05:42 MST 2017
> >>dera...@armv7.openbsd.org:/usr/src/sys/arch/armv7/compile/GENERIC
> >> real mem = 536870912 (512MB)
> >> avail mem = 517271552 (493MB)
> >> mainbus0 at root: Olimex A10-OLinuXino-LIME
> >> cpu0 at mainbus0: ARM Cortex-A8 r3p2 (ARMv7)
> >> cpu0: DC enabled IC enabled WB disabled EABT branch prediction enabled
> >> cpu0: 32KB(64b/l,4way) I-cache, 32KB(64b/l,4way) wr-back D-cache
> >> sxiccmu0 at mainbus0
> >> simplebus0 at mainbus0: "soc"
> >> sxipio0 at simplebus0: 175 pins
> >> sxitimer0 at simplebus0: cntrtimer @ 24000KHz
> >> sxie0 at simplebus0, address 02:02:0a:81:e6:6c
> >> rlphy0 at sxie0 phy 1: RTL8201L 10/100 PHY, rev. 1
> >> sximmc0 at simplebus0
> >> sdmmc0 at sximmc0: 4-bit, sd high-speed, mmc high-speed, dma
> >> ehci0 at simplebus0
> >>
>
>
> Thanks for the reply Artturi!
>
>
> You are right, I did not notice the tick/stattick values.
>
> Regarding sxie.c, I've gone ahead and yanked out the second TX FIFO, used
> ifq_restart in the ISR (as per example in ifq.h) and increased send queue max
> length from 1 to IFQ_MAXLEN. Now I'm getting a much more healthy 1MB/s TX
> and ~8MB/s RX, even when doing both.
>
> If anybody more knowlegable thinks it's worth it and would like to give some
> feedback on the diff, I would be more than happy to clean it up and submit to
> tech@. I agree though that giving up on the second TX FIFO might be just
> side-stepping the problem.
>
>
> Regards,
> Eduard
>
I was going to take a look at this too, but got sidetracked before i got
anything finished, so i appreciate your efforts :)
fwiw. i made some notes while digging for better docs/understanding:
- apparently NetBSD never got the second fifo working:
static void
awin_eth_txfifo_transfer(struct awin_eth_softc *sc, struct mbuf *m, u_int slot)
{
bus_size_t const io_data_reg = AWIN_EMAC_TX_IO_DATA_REG(0 /*slot*/);
and most issues i came across googling were on linux/receiving side,
so i think the TXFIFO can certainly be made to work.
i know 100% i had it totally working at some point, maybe never on CVS,
as there's no traces of DMA there either:)
i found enough evidence to assure myself, that we're actually dealing with
PE-MACMII by Mentor, which has all the pieces we're playing w/like
AHB DMA Ethernet bridge and FIFO Memory interface etc..
originally this was thought to be related to dm9000 or something, but maybe
it was just what the first Allwinner source was based on, and later authors
just kept passing it on as such...
so this could be very much like the EMAC in LPC23XX[1] w/different vendorglue.
might not be waste of time to read some user manuals by NXP covering the EMAC.
-Artturi
[1] http://www.keil.com/dd/docs/arm/philips/lpc23xx.h
>
> Index: sys/arch/armv7/sunxi/sxie.c
> ===================================================================
> RCS file: /cvs/src/sys/arch/armv7/sunxi/sxie.c,v
> retrieving revision 1.25
> diff -u -p -r1.25 sxie.c
> --- sys/arch/armv7/sunxi/sxie.c 22 Jan 2017 10:17:37 -0000 1.25
> +++ sys/arch/armv7/sunxi/sxie.c 19 Nov 2017 00:06:19 -0000
> @@ -97,7 +97,7 @@
> #define SXIE_MACA2 0x00a0
>
> /* i once spent hours on pretty defines, cvs up ate 'em. these shall do */
> -#define SXIE_INTR_ENABLE 0x010f
> +#define SXIE_INTR_ENABLE 0x010f
> #define SXIE_INTR_DISABLE 0x0000
> #define SXIE_INTR_CLEAR 0x0000
>
> @@ -106,7 +106,7 @@
>
> #define SXIE_RX_ENABLE 0x0004
> #define SXIE_TX_ENABLE 0x0003
> -#define SXIE_RXTX_ENABLE 0x0007
> +#define SXIE_RXTX_ENABLE 0x0007
>
> #define SXIE_RXDRQM 0x0002
> #define SXIE_RXTM 0x0004
> @@ -250,7 +250,7 @@ sxie_attach(struct device *parent, struc
> ifp->if_watchdog = sxie_watchdog;
> ifp->if_capabilities = IFCAP_VLAN_MTU; /* XXX status check in recv? */
>
> - IFQ_SET_MAXLEN(&ifp->if_snd, 1);
> + IFQ_SET_MAXLEN(&ifp->if_snd, IFQ_MAXLEN);
>
> /* Initialize MII/media info. */
> mii = &sc->sc_mii;
> @@ -440,16 +440,10 @@ sxie_intr(void *arg)
>
> if (pending & (SXIE_TX_FIFO0 | SXIE_TX_FIFO1)) {
> ifq_clr_oactive(&ifp->if_snd);
> - sc->txf_inuse &= ~pending;
> - if (sc->txf_inuse == 0)
> - ifp->if_timer = 0;
> - else
> - ifp->if_timer = 5;
> + ifq_restart(&ifp->if_snd);
> + ifp->if_timer = 0;
> }
>
> - if (ifp->if_flags & IFF_RUNNING && !IFQ_IS_EMPTY(&ifp->if_snd))
> - sxie_start(ifp);
> -
> SXISET4(sc, SXIE_INTCR, SXIE_INTR_ENABLE);
>
> return 1;
> @@ -468,42 +462,36 @@ sxie_start(struct ifnet *ifp)
> uint32_t fifo;
> uint32_t txbuf[SXIE_MAX_PKT_SIZE / sizeof(uint32_t)]; /* XXX !!! */
>
> - if (sc->txf_inuse == (SXIE_TX_FIFO0 | SXIE_TX_FIFO1))
> - ifq_set_oactive(&ifp->if_snd);
> + if (ifq_is_oactive(&ifp->if_snd))
> + {
> + printf("sxie: start called while in use\n");
> + return;
> + }
>
> - if (!(ifp->if_flags & IFF_RUNNING) || ifq_is_oactive(&ifp->if_snd))
> + if (!ISSET(ifp->if_flags, IFF_RUNNING))
> + {
> + printf("sxie: start called when not running\n");
> return;
> + }
>
> td = (uint8_t *)&txbuf[0];
> m = NULL;
> head = NULL;
> -trynext:
> - m = ifq_deq_begin(&ifp->if_snd);
> +
> + m = ifq_dequeue(&ifp->if_snd);
> if (m == NULL)
> return;
>
> if (m->m_pkthdr.len > SXIE_MAX_PKT_SIZE) {
> - ifq_deq_commit(&ifp->if_snd, m);
> printf("sxie_start: packet too big\n");
> - m_freem(m);
> - return;
> + ifp->if_snd.ifq_errors++;
> + goto end;
> }
>
> - if (sc->txf_inuse == (SXIE_TX_FIFO0 | SXIE_TX_FIFO1)) {
> - ifq_deq_rollback(&ifp->if_snd, m);
> - printf("sxie_start: tx fifos in use.\n");
> - ifq_set_oactive(&ifp->if_snd);
> - return;
> - }
>
> - /* select fifo */
> - if (sc->txf_inuse & SXIE_TX_FIFO0) {
> - sc->txf_inuse |= SXIE_TX_FIFO1;
> - fifo = 1;
> - } else {
> - sc->txf_inuse |= SXIE_TX_FIFO0;
> - fifo = 0;
> - }
> + ifq_set_oactive(&ifp->if_snd);
> +
> + fifo = 0;
> SXIWRITE4(sc, SXIE_TXINS, fifo);
>
> /* set packet length */
> @@ -518,15 +506,14 @@ trynext:
> /* transmit to PHY from fifo */
> SXISET4(sc, SXIE_TXCR0 + (fifo * 4), 1);
> ifp->if_timer = 5;
> - ifq_deq_commit(&ifp->if_snd, m);
>
> #if NBPFILTER > 0
> if (ifp->if_bpf)
> bpf_mtap(ifp->if_bpf, m, BPF_DIRECTION_OUT);
> #endif
> +end:
> m_freem(m);
> -
> - goto trynext;
> + return;
> }
>
> void
