I am looking for a discussion about wrapping Hugh table as collection

Hi,

I am looking for a discussion about wrapping Hugh table as collection

I read a lot of material regarding O/R mapper and I could not find a
discussion about
wrapping Hugh table as collection while preserving object identity and COM
rules

More:

I need to create an ATL collection to use in VB which wraps a Hugh data base
table
I am familiar with the Iterator pattern but i dont want to introduce new
syntax to VB programmers

In order not load the entire table to memory i came with the following
solution:

1.Data mapper layer which can load a page(using ado paging) of objects or a
single object using criteria,into a Cache
2. a Cache which hold loaded objects,
the cache itself built from two list,one(SEQUENTIAL) which hold all
objects which belongs to the current loaded page,
and the other(IMMEDIATE) contains all objects which have(COM)
references and are not in the SEQUENTIAL page

In general what I do is the following

When a new page is need to be loaded,all object from the SEQUENTIAL(the
current loaded page) which have References are moved to the IMMEDIATE
list,those with out references are free from memory.

Next ,for each loaded row I check if the IMMEDIATE contains an object for
this row
if so,I moved the object from the IMMEDIATE list to the
SEQUENTIAL,otherwise I create a new object
and put it into the SEQUENTIAL list
In such way I can ensure only single object for a row is exist in memory

when adding new objects(Collection.Add(...)) it added to the IMMEDIATE page.

When removing an object I need to calculate it's relative index position to
the current loaded page
(in page,before page,after page)
in order to adjust the indices(currently I just thinking to refresh the
loaded page)
My system is for desktop application,single user.

for example

N=100

dim colItems as Items
dim a as Item 'Key of this item is 'itemA,Index=5000"
dim b as item 'Key of this item is 'itemB,Index=6000"
dim c as Item
dim d as item

set a=colItems(5000) ' 1.will load row 5000 to 5099 into SEQUENTIAL list

set b=colItems(6000) '1.need to load new page so move 'itemA' to the
IMMEDIATE list because it has reference
'2.free the SEQUENTIAL list
'3.load row 6000 to 6099 into SEQUENTIAL

set c=colItems("itemA") 'already in memory( IMMEDIATE list,index=5000)
return a reference to the item
'a and c both point to the same object
now


set d=colItems(5000) '1.need to load new page so move 'itemB' to the
IMMEDIATE list because it have reference
'2.We already have the 5000 item('itemA')
in IMMEDIATE list,so move it to the SEQUENTIAL
'3 create objects for rows 5001-5099 and
add them to the SEQUENTIAL list
' 'a and c and d are all point to the same
object now

in general my system use key to access individual objects and index when
displaying a list

Thanks.