this is for iOS (actually tvOS) but the comparison should be identical.
I have a data file that is CSV, it is about 22K in size and consists of 300 lines with a varying number of fields on each line.
During the execution of the program, at various times it needs to access ONE of those lines (the index will be determined based on current state of the app , from 1 to 300)
The question is which is a better way to access
Read the data as a textfile each time a line required, closing the stream when the desired line is read
Read the entire file into an array, and access directly using index?
Speed is not a huge factor. So method #1 saves on memory, seeing as the file resource is in the App RAM space already, while method #2 might be “faster”, but doubles the memory required as two copies of the data would exist (one in the App Resource, and another in the Array)
The data is immutable, so it isn’t going to be manipulated, and written again
I’m leaning towards method #1, but would like opinions ,
Before anyone mentions it… SQLite is NOT an option (for iOS it would be, but not for tvOS)
well #2 isnt “double the memory” is it ?
its a copy on disk IN the app on disk, and one in memory
the second has slightly inflated requirements as you read (but not as much as the first)
the data seems small enough it might not be that big a hit to read it once & just hold it in the array
I would read the content in an array and use the array as lookup table. It costs a bit of memory but the solution of reading the file always when needed new would result in the same cause it would read always when data are needed. In my eyes a lookup table is a more comfortable and less tricky method and it would need the loading of the file once at the beginning.
In case of the Memory: we speak about 22kbyte of memory which we use more over the entire runtime. I can’t see that this would be a showstopper with this memory usage. On the other side we will use the memory anyhow when reading the file. As far as I know Xojo fw has no chance for removing the variable from memory means we would have also there a memory usage in that height or less (depending on the line number I need to catch up.
no it is not a copy “on disk” since the ATV and iOS devices don’t have a disk… it is all RAM
I opt’d for another alternative… it reads the “file” as a string (similar to how Xojo does it) and then splits the string to get to the desired components. Since the string is scoped local to the function, its memory hit is minimal, and lifespan is short
Bit the result is what I said. Not from Disk but virtual File System in memory. But reading the file as String and builsing a lookup table (means an Array) via split. Can be done in both, Xojo and also Swift.
I will play devil’s advocate. Since you point out that files take up memory in their own right on such devices, one could read through and discard the entire file, building a compact array of pointers to the start of, and length of each record, probably along with a key. Given that the file is now open, and given a key, one could seek to the offset, grab the string and extract what you need from it.
That would not be that hard to code, but it is more complex and a higher cognitive burden than either of the other methods mentioned. And heck it is just 22K in size.
Also it would be a lot more complex if this isn’t a static data file, at least during the run of the program.
May I make a suggestion for an alternative, that would be very performant and memory efficient?
Use a binary format, with a table in the header, comprising of Uint32 or Uint64. This table simply points to where the actual data starts.
Your app reads ( Uint32 * Index ) to get the offset, moves to that location, reads another Uint32 to get the length, then reads the data and closes the stream.
This is a lot more work for you, especially as you’ll need to build an app to create the binary file, but it does mean that the data doesn’t need to be duplicated and it doesn’t matter which index the app needs, it will always take the same time to read it, 3 file reads.
