Database creation and file browsing – Request for comments
This is a little call for feedback. In order to simplify the general usage of file lists – such as ROM lists, cheat lists, skin lists… – in terms of programming I would like to generalize directory access.
The sd2snes currently scans the entire card on startup and rebuilds an internal database if any change to relevant card contents is detected. This takes place so the file browser can always show the current state on card. Said database contains the entire directory structure and takes up quite some “ROM” space. I would like to replace this “deep scanning” by a simple “list a single directory” function which can also be used to list other kinds of files under different circumstances. This would mean that a folder will be loaded on the fly when you open it, instead of all folders being preloaded at startup.
The current state of implementation has one advantage: browsing the card is fast. There are no delays when changing directories.
The downside is that you have quite a lengthy “Loading …” screen at bootup because of the full card scan. And if something is added/removed/renamed the database rebuild can take a long time.
The advantage of proposed on-the-fly file list creation is that there is almost no loading time at startup. I could even ax the “Loading …” screen altogether. There will be no lengthy “rebuilding database …” either every time you change something on card. Files could easily be manipulated on the fly, too – some people asked for a file deletion feature – without having to rebuild the entire database. Also the database creation code is a complex mess and I want to get rid of it… 😉
The disadvantage is that there will be slight delays while browsing (as a folder is read and sorted on demand) – if a single folder contains several thousand files this can amount to a few seconds.
So um, would you mind if I changed file handling to the latter?
I didn’t mind the lengthy loading screen, but if you prefer that kind of loading on the SD card, I’m fine with that.
Don’t see a problem with making the change…..
Does the typical user have 100s of Roms or do they generally have less? I think you might have a mix of both so in each use case there would be a better option. Maybe add a variable that can be changed in a config file to allow for the old database mode. Choice is best.
Hey ikari!
I personally don’t mind at all the startup delay (I do have to go sit and grab the controller after powering up the SNES), but I understand the motivation for the simple directory listing. I say go ahead! 🙂
Since I’m changing the files on my SD card very often I definitely prefer the second version.
Also: From my point of view normal people don’t have several thousand files in just one folder, therefore it should not make much difference in the user experience but for developers who want to participate in the sd2snes code it is good to have a simple and understandable directory listing code.
Initially quicker is much better, love the idea.
I personally like the way it is right now, there’s no need for a change..
That’s just my opinion though. 🙂
I don’t change the contents of the card and I have thousands of roms.
My cartridge starts up immediately and the navigation is a blast, even in folders with hundreds of roms.
For me this faster navigation within folders is a feature compared with other sd based cartridges.
I’d keep it as it is now or, at the very least, leave it as an option.
I’d prefer the “load on the fly” option 🙂 I cannot really explain why, but I love the idea… However, I don’t have that much ROMs on my SD-card.
Having all my ROMs organized into regional and alphabetical folders, I like how it works now. No delays anywhere, really. I’m wondering if the loading at startup is more a psychological thing. Most SNES games don’t boot right into action, either. So maybe replacing “Loading…” with a logo intro screen could alleviate some of the frustration that comes from the delay?
As for a delete feature, I’m not a huge fan of the idea. But if you put one in all I would ask is that it’s not something that could easily be done by accident.
Interesting thought! 🙂
You’d do all directory scanning using the sd2snes’ dedicated hardware, I presume? (Just curious because I’ve ever been struggling with the 65816 speed bottleneck on the PowerPak …) If so, then I’d recommend to go ahead and scrap the database approach in favor of speedy on-the-fly directory loading. 😀
Thanks for your ongoing support! 🙂
Ramsis
Yes, definitely. The database is also generated using the on board MCU, no SNES CPU work is involved. The SNES has no direct SD access anyway. 😉
Hey Ikari, nice to see you at work 🙂
I think you can do that kind of change,
I use to have roms in folder (platformer / rpg / sports…)
i don’t think it will change the navigation, and as juef say
I’m in my sofa with my controller in my hand, no problem for little delay
BUT i have a request
can you implement a “FULL RESET” command on SD2SNES Menu ?
when i’m on my sofa, i dont want to move 🙂
Thank you for all Ikari
Seriously !
How about an option where you can select the load method?
Well I’m trying to tidy up the code and make development easier. Adding more bloat won’t help much with that 😉
Then I would prefer the new method.
I think that most SD2SNES users have sorted their files on the SD card.
Then you have room for other features. 😉
I’d rather have a lengthy start up time than have a l slow browsing time myself. Keep it more or less like it is.
It would be nice if you added a favorites folder (I know I can make my own) with an on the fly “add to favorites” option to the menu, but that’s probably a lot of work.
I’m about to order a SD2SNES, but I would say that having a on-the-fly loading would be nice. I have yet to see how the list is done on the SNES itself – does the firmware inspect the ROM header to show the title of the game, or does it show the file name? If it’s the former, than I can see that on-the-fly will bog down with a lot of ROMs. Again, it’s up to the dev(s) to see what setup works the most efficent, but on-the-fly would be nice. At least just have a hourglass as the list… Read more »
The firmware only scans the file name and file size and displays those. Inspecting the ROM headers would be prohibitively slow even when generating the database in advance.
Hi Ikari, I like the actual implementation with deep scan and indexdb : i have severals sd cards with roms organized by regional, years, or genre… and one (little, an old 2Gb sdcard) with roms i change on the fly (trads, patchs, homebrew…). So I (and it’s only my opinion, no offense to anyone) prefer the actual implementation with an indexdb, even if it takes a lot of time if i change a file on a 16gb sdcard :p Most of the time there’s no change on the huge sdcards. But i understand you want to get rid of this… Read more »
Thank you so much! I really appreciate it! 😀
Do it.
I think the best course of action would be to implement it as an option in the options menu, seeing as this has about as equal support on both sides. I already like the way it is, I rarely make any changes to my romset, so my database stays pretty solid all the time and I don’t have to be rebuilding it every time I boot it up. And I really love the feeling of browsing so quickly between different folders when I’m not sure what to play, I wouldn’t want that to be taken away by any means. So… Read more »
Go for it! Clean out that crufty indexing code 🙂
I have a feeling that the delay is not as bad as some envision it. I think I’m going to prepare a demo firmware so you can try it yourself. 🙂
I have the snes fullset and I sorted by continent (Pal, Jap and US), there are about more or less 1000 rom snes by directory.
Made the wait does not bother me, especially since it is fast enough, I prefer to navigate the repertory rom without slowing in my case.
Personally, I say go for it. I’m happy either way as long as I can get my SNES fix.
I like the current method, I prefer a slow boot time than slow directory browsing. I’ve got all games ever released on my SD and never change the contents, so one big slow Loading screen is no problem for me.
Maybe you could provide a tool for PC (maybe coded in Java, so that it works everywhere) which simply generates the database on a fast machine and saves it on the SD card. So no long waiting time on the sd2snes if you change anything.
Thanks and regards
i like both methods ( because you only play at 1 game at time , if you spends more time on menus that playing i reccomend use a emulator on pc xD ) but any upgrade that is made for the SD2SNES is very and likely welcome .
and is posible do a maker of themes for the sd2snes menu?
and support for covers?? ( in bmp , jpg , tiff , or anything less recurse consuming)
Both things hav etheir distinctive ups and downs. Changing the conten of your card is mostly a thing for smaller sized cards or does happen rather rarely (If a new hack / translation shows up or if you discover an obscure new game) but seeing how cheap SD cards have become, the former shouldnt be a problem. With people reporting, that it works even with a full rom set (the whole RAM thing) and its still comfortable to use that way, i dont see a problem from a technical standpoint. On the other hand, im not a programmer, so i… Read more »
I’d rather have the initial load time and be done with it than having browsing slowed down, so I personally would like it if you leave the current system in place.
Remove the cruft (and obviously an option for it is the worst scenario for everybody)!
Another question: do you currently have anything planned for the USB port? We’re developing a custom piece of hardware that goes into the joypad port, so testing on emulators is not possible and with the current slow (copy build to SD card, eject, insert into SD2SNES, boot up, select file) it feels like that USB sits there with a smug smile every time. (:
Some sort of developer mode over USB would be extremely awesome!
I’m on it 🙂 First results show that a directory scan, including sorting, takes about 40 milliseconds for a directory with 100 entries, and about one second for a directory with 1000 entries, which seems ok. When disabling sorting everything shows up instantly but I’m definitely not a fan of having to use impractical FAT sorting tools and the like… I could agree with adding an option to disable directory sorting though. As for USB, yeah I have something in mind for that but the implementation details are still foggy… I’d like to provide an upload/configuration interface as well as… Read more »
I think it’s a good move. It makes no real difference to me, personally, because I don’t load my SD2SNES with a crapton of ROMS – I only really put the ones I actually want to play on it, and I have them separated by name in alphabetic folders. With a prospective speed of one second per 1000 entries in a directory, I can’t really see any practical downside.
Honestly – how many of you have a thousand files on the thing in TOTAL, let alone in a single directory?
I have a full ROM set and a full SPC set on the card myself for testing purposes. Around 40000 files total, in 1500 directories! So whenever I put a new file on card – “rebuilding database, please make some coffee” 😀
I have at least 5000 roms myself that’s games from multiple regions, translations and home brews but I realize I’m probably in the minority though..
I also have thousands of files organized in around 100 directories but I don’t see a performance issue; I like it the way it is.
Ikari, after all theses changes will you consider adding compatibility to more special chips in the future? I mean sa-1 and super fx….
Certainly!
I like how it currently is. To be honest I don’t even notice the “Loading…” screen. It’s usually gone by the time I travel from the snes to my chair, after powering up. I guess it depends on how much browsing would be slowed down. either way, I ♥ my sd2snes so much.
Hey Ikari, How close are you from having 100% compatibility? How many firmwares or hardware incarnations are we from having every game working perfectly? Just take your best guess.
Good answer.
Please do not put me under pressure. Last time I took that to heart (among other things) I became unable to do any development for an entire year. You seem to imply I didn’t reply to your comment on purpose which is disheartening and demotivating. In fact after reading your question I thought that it would require some effort to answer. Deciding to think about it a little more I eventually forgot about it. Sorry. You are not the only one. Moreover I can’t spend all day in front of my inbox waiting for comments to reply to. sd2snes will… Read more »
Thank you for answering my question the best you could.
I’m just glad I don’t have to spend a small fortune to play the games I played as a kid. As close to the the way they were meant to be played as possible.
sd2snes is probably the way I’ll introduce my kids to video games. ..
I should’ve realized I’m not the only one clamoring for you’re attention about the project on this site and other places.
I didn’t mean to upset you, if I did, I’m sorry. Keep up the good work.
Please leave it as it is!
Hi, I currently keep 3000+ roms (all of the official roms plus a small amount of patches) on my card in about 90 folders. I’m constantly adding new roms because I’m developing randomizers for a couple of SNES games. The database rebuilding at the beginning doesn’t bother me because it’s actually quite quick, even with this many files. I’m also a fan of how fast the browsing is. I think my preference is to leave it as is, but since I haven’t seen how your new setup will work exactly, I don’t have a basis of comparison. If it makes… Read more »