[Yaffs] YAFFS2 Development

[Charles Manning]

Hi Jon

It is always great to get a bunch of feedback from someone doing something a 
bit different and stressing the system.

Some of the issues you raise have, I think, been dealt to in the last while, 
or have been recently tackled but there is still scope do use this info in a 
very useful way.

I have some comments below. I would appreciate any further comment/patches as 
this will allow the code to be improved. I am not at all obsessed with the 
form it is provided in (C, patch, whatever). Someone that has been through 
the code as thoroughly as this will have a lot of useful perspective that I 
hope to be able to exploit :-).




On Wednesday 07 December 2005 05:10, Jon Masters wrote:
<snip nice words>

> * YAFFS2 should not use any shortopcaches when on Linux. It seems to
>   make little difference to performance (only negative).

That was my initial impression too. The shortop cache stuff was originally 
written for WinCE which does not work through a page cache like Linux does. 
In paticular nasty situations like 

//Yes, real WinCE programs really have code like this
 while(!eof) {
    read one byte
 }


YAFFS was dead without some caching. By default it was turned off for Linux. 
Then somebody ran code of the form

  for(i = 0; i < 100000; i++) {
    write one byte
  }

The Linux cache is write-through so these calls were observed to be slow under 
Linux too and enabling the short op cache fixed the problem. From then on, 
the shortopcache has been enabled by default.

Thoughts:
1) Has the Linux page cache changed so that this is no longer an issue?
2) Perhaps only using the short op cache for write operations would be the 
best way to do things under Linux?

>
> * YAFFS2 memory allocation using kmalloc does not work on very large
>   devices and needs to use vmalloc instead in those cases (>2GB devices).
>   The lack of checking for success proves to be a problem.

I think this only impacts on the creation of the huge chunk bitmap structure. 
If so, this was dealt to in  
http://www.aleph1.co.uk/cgi-bin/viewcvs.cgi/yaffs2/yaffs_guts.c?r1=1.20&r2=1.21
Andre tested this, IIRC, and this fixed the problem.

Is more required?

Yes, definitely the handling of alloc failures is a bit sloppy.

>
> * YAFFS2 has various internal usage of types which makes it difficult to
>   scale to >2GB devices. We have to divide up into multiple partitions.

Can you give some details? I would like to fix this. There are some places 
where where ints are being used where off_t would be correct.

The chunkGroupBits issue also has impact on this.
>
> * Andre Renaud latched onto a problem which I then rediscovered in
>   performance testing. Having chunk groups of 16 reduces performance by
>   at least 50% but in practice can be much higher. By applying a version
>   of his patch, I was able ot reduce read time for a 50MB file from 27
>   seconds to around 15 seconds and have achieved sustained reads at
>   22.2Mbit/s on multi-GB devices reading many hundred MBs.

I have written some code (minor testing so far, more testing and checkin 
within 24 hours I hope) which should fix this.

This code uses variable size bitmaps to fit the required bit width, thus 
eliminating chunkgroups, but does not use as much RAM as the Bluewater hack.

>
> * YAFFS2 makes use of some additional reads and memcpy's which don't
>   seem entirely necessary - by combining and changing some of the logic
>   it looks like we could get another 10% performance gain.

Very much look forward to more info on this.
The WinCE stuff has some extra copying (that is actually no longer required 
and will be eliminated). I hoped the Linux stuff was not doing too much exta 
work.

-- CHarles