| |
Perff
Administrator
Posts: 1665 |
CSDb V2
As some of you might have seen, there have been talked a bit about a CSDb V2.
I have also had this idea for some years now, because of some basic flaws in the originally database design of CSDb.
Now is the time to try to do something about it.
But we, the current CSDb staff, don't have the resources to do this, so we need your help!
Here is what we need:
1. Somewhere to host CSDb V2. The current hosting for CSDb is not very good, as some of you might agree on, and it would be very difficult for more people to develop on it, so a new hosting place is needed.
The requirements for such place are not to much. Talking about performance, CSDb can run on any normal computer today with no problem, and a few gig of disk space should be ok. The daily trafic sums up to about 40-50.000 hits and 300-400mb.
Some kind of CVS or similar to make it easy for multiple people to develop on it, is also required. (PS. We got no money! )
2. People that would help. First of all we need some willing developers - PHP & SQL. I don't think we should get too many - 2-3, perhaps 4 (plus myself :) ) We'll figure that out.
Also some design people, to make the site look kewl. :) And finally some people with deep knowledge of the scene so we are sure to get around every corner in the design phase.
3. Time and patience. :)
The plan is then to put togehter a CSDb V2 team (not to big, 6-8 people tops), and figure out exatcly how to make it. Of course we should look at the current CSDb, but I sugest we make everything from scratch.
Most importaint is the design phase. We must try to take all into account when designing it. I think this is the most importaint part of it all.
Then it gets coded, designed etc, and when it is ready for release, we transfer all the data from CSDb to CSDb V2. :)
Even though the CSDb team is the ones who should make it, it dosn't mean that the rest of you have nothing to say. Perhaps we could post our plans somewhere for all of you to comment on.
That is roughly how I see it.
Now lets have a nice little discussion. :) |
|
| |
Nafcom
Registered: Apr 2002
Posts: 588 |
But you will take over the data from the current CSDb which people entered data into over the recent years, or? |
| |
Perff
Administrator
Posts: 1665 |
as i wrote:
Quote:and when it is ready for release, we transfer all the data from CSDb to CSDb V2.
So yes. :)
It's the idea that the general user one day goes to the current CSDb, and the next day to V2 without any data lost. |
| |
Count Zero
Registered: Jan 2003
Posts: 1821 |
Well, I wouldnt vote for that, since all the data entered here is wrong anyhow. :)
The hosting, resulting traffic and proper resources shouldn't be problem - you have been offered to move to the pokefinder.org servers whenever you want. :)
l8r
Count Zero/CyberpunX/SCS*TRC |
| |
iopop
Registered: Dec 2001
Posts: 317 |
Dont know if we should answer here if we want to help. But Im willing to do php/postgres code. |
| |
TDJ
Registered: Dec 2001
Posts: 1879 |
And since I'm in the business of thinking up flexible systems anyway, I'd like to volunteer for the design phase. |
| |
Perff
Administrator
Posts: 1665 |
I'm glad to se that there are people outthere that will help make this happen. Thanks. :)
But we could still need some with some expertice in designing webpages. (html etc.)
I'm not very good at making a website look fansy. :)
Anyone?
|
| |
T.M.R
Account closed
Registered: Dec 2001
Posts: 749 |
i'm tempted to offer for the design side, but only if it's a last resort 'cos A) there are far better than me around and B) i quite like the way this site looks at the mo! =-) |
| |
Hein
Registered: Apr 2004
Posts: 933 |
Quote:
Well, I wouldnt vote for that, since all the data entered here is wrong anyhow. :)
The hosting, resulting traffic and proper resources shouldn't be problem - you have been offered to move to the pokefinder.org servers whenever you want. :)
l8r
Count Zero/CyberpunX/SCS*TRC
I think this scene is critical enough to change/point fingers whenever there is wrong data. It suprises me it's not the mess it could have been.
But if you're volunteering to add all 30000 something entries again, with correct data, even if it allready is correct data. |
| |
Nafcom
Registered: Apr 2002
Posts: 588 |
Quote: I think this scene is critical enough to change/point fingers whenever there is wrong data. It suprises me it's not the mess it could have been.
But if you're volunteering to add all 30000 something entries again, with correct data, even if it allready is correct data.
Totally my opinion! :) |
| |
Mirage
Registered: Jan 2003
Posts: 113 |
So it's settled then? count zero is going to do everything... that's pretty cool, because then we wouldn't need that CVS system for multiple people... he'll just be doing all the screwing-up and we (not me, ofcourse) will know where to point our fingers at :) |
| |
V-12
Registered: Nov 2003
Posts: 206 |
I see no sense for doing something from the beginning again, better way to improve an existing CSDb. It would be a waste of time for woking ong csdb v2
|
| |
Perff
Administrator
Posts: 1665 |
@T.M.R: No one said that the design should be very much different from how it looks today. But if we start from scratch, the html will also need to be made, and the one who created the originally design is no longer active. Also the html-standart have changed a bit over the last 4 years, so there might be better ways to set up a page, than just doing tables within tables within tables... (as CSDb is today) :)
@Murdock: Perhaps you are right. However you guys keep on coming with ideas for improvement of CSDb (which is nice :) ), but as the lone coder of CSDb there is no way I can do it all. Also the things I have made are mostly hacks on hacks because on flaws in the originally DB-design, so even if I got some extra coders to develop on CSDb V1, it would be a nightmare for them to understand and further develop it.
This has been the fact for a long time now, so if CSDb should implement all or most of the wishes you have made, a V2 is required! |
| |
Hein
Registered: Apr 2004
Posts: 933 |
Tables inside tables inside tables is still a solid technique and has become a real standard. XHTML 1 is supposed to be a standard, but every obscure (fill in your favourite nemesis) browser interprets it different. I would stick with the tables inside tables. Besides, this is an 8 bit community, relying on old techniques.
Maybe the graphical design can be retouched a bit, as can the way entries are added. Somehow it takes too many steps to add an entry. |
| |
TDJ
Registered: Dec 2001
Posts: 1879 |
1. Appearantly tables inside tables is the devil, and <div> tags are the way to go. At least that's what I learned in the 'real' world ;) But yes, there may be some problems with compability.
2. CSDB v2 is a good idea, hacking CSDB v1 further into something that will collapse under its own weight is bad, hmmmm. Murdock, remind me never to offer you a job as a web-application-programmer, okay?
3. You shouldn't just change the way things look, the most important is the user interface. Make it easier for people to add stuff etc. like Hein proposed. Interaction Design is a must these days! |
| |
Hein
Registered: Apr 2004
Posts: 933 |
TDJ, I was about to tell the world you are a great software architect, but you proposed to help yourself allready. ;) |
| |
T.M.R
Account closed
Registered: Dec 2001
Posts: 749 |
Well, scream if y'want me Perff but consider me a fallback option - if anyone else wants a go, give them the gig over me. i'm a tables-in-tables sort of person, for reference. =-)
Oh, whilst i'm here... why do most of my entries list me as T.M.R/Onslaught rather than Cosine...? |
| |
Hein
Registered: Apr 2004
Posts: 933 |
Quote: Well, scream if y'want me Perff but consider me a fallback option - if anyone else wants a go, give them the gig over me. i'm a tables-in-tables sort of person, for reference. =-)
Oh, whilst i'm here... why do most of my entries list me as T.M.R/Onslaught rather than Cosine...?
Provide join/left date for Cosine.
Maybe it should be the current group(s) by default, instead of groups with dates set. |
| |
T.M.R
Account closed
Registered: Dec 2001
Posts: 749 |
Ta Hein, although now it's listing me as Cosine and Onslaught for Cosine stuff! =-) Might be nice to have an option in V2 to select which group(s) the handle was working for at the time...? |
| |
Hein
Registered: Apr 2004
Posts: 933 |
yeah.. mmm, shit, complicated.. bleh, bleh, go away, complicated thoughts..
probably allready works that way, only the release will need a releasedate as well.
oh, wait.. set a join date for Onslaught.. :) maybe that works.. minute accurate, please.
Glad count zero will fix all the info, sigh.. he's da man. |
| |
Count Zero
Registered: Jan 2003
Posts: 1821 |
Yeah, I will fix EVERYTHING and lock it up properly then, so nobody can change my book on scene history. :)
However, table in tables in etc. are NOT the way to go for sure. Especially since C64 browsers handle that so badly. :)
l8r
Count Zero/CyberpunX/SCS*TRC |
| |
Hein
Registered: Apr 2004
Posts: 933 |
Then what are you proposing? :) |
| |
chatGPZ
Registered: Dec 2001
Posts: 11108 |
while you are at it, would you fix my income too? thanks in advance :o) |
| |
Steppe
Registered: Jan 2002
Posts: 1510 |
Quote: Then what are you proposing? :)
The beauty of CSS: http://www.csszengarden.com/ |
| |
Hein
Registered: Apr 2004
Posts: 933 |
aah.. css. bah, but I do that at work too, boring.. :) |
| |
Nafcom
Registered: Apr 2002
Posts: 588 |
I have another idea for CSDb V2. Remove all maintain permissions, except:
-Your own profile
-Your group
-Your products
So the key words are "reset permissions" |
| |
Steppe
Registered: Jan 2002
Posts: 1510 |
And make all votes transparent, while you're at it... |
| |
Nafcom
Registered: Apr 2002
Posts: 588 |
Quote: And make all votes transparent, while you're at it...
And I am still against it. I do not want that anybody can see for what I vote :) |
| |
Oswald
Registered: Apr 2002
Posts: 5017 |
transparent votes is a must against downvoting. check out poeut, it works there, and noone has anything against it.
if you cant stand for your opinion then dont vote. period.
lets stop the childis downvoting game.
Nafcom: why do u need the privacy, if not for hiding all your 1's ? eh ? |
| |
Nafcom
Registered: Apr 2002
Posts: 588 |
Quote: transparent votes is a must against downvoting. check out poeut, it works there, and noone has anything against it.
if you cant stand for your opinion then dont vote. period.
lets stop the childis downvoting game.
Nafcom: why do u need the privacy, if not for hiding all your 1's ? eh ?
never voted any production a 1. :)
But I am sure the comments like "lamer, etc" would rise then. Because I think I have some 2 or 3's.
Hmm, on Pouet, the voting system is different, one only has thumbs up, down or a piggy :) |
| |
TDJ
Registered: Dec 2001
Posts: 1879 |
Quote: never voted any production a 1. :)
But I am sure the comments like "lamer, etc" would rise then. Because I think I have some 2 or 3's.
Hmm, on Pouet, the voting system is different, one only has thumbs up, down or a piggy :)
Please don't start this discussion again, not in this thread. There are 1,047 threads already about this subject here ;)
|
| |
iopop
Registered: Dec 2001
Posts: 317 |
I say we skip the voting completely in v2.. :) |
| |
Oswald
Registered: Apr 2002
Posts: 5017 |
belongs here if we want csdb v2 have transparent votes. If csdb v2 will be reality, another voting should be done about this.
To add to the topic, I think that the user interface here is not very friendly. The design is ok. The look and feel overall is good. But when it comes to doing something, adding a product, searching, etc I found that I have always look through the whole screen in the search of where to click. What I try to articulate here is that the gui is everything but NOT intuitive. It feels for me that every button, info/text is on the wrong place.
For ppl that already added 100 entrys this is probably not an issue, but guess one needs quite some excersize to use the system from "instinct" without stopping and let your eyes go around where to click next.
(example: I had no idea for years (months? when was this added?) that what the "PM" button does here in the forums, does it really worth the -317 bytes not to have it say "Send Private Message"? we are a bit too much on the purist side) |
| |
T.M.R
Account closed
Registered: Dec 2001
Posts: 749 |
Well, the acronym PM is in fairly common use and "Send private message" would be quite a large button... perhaps "Send PM" would be a compromise and an alt tag that pops up the full "Send private message" maybe...? i do agree that the system could be more intuitive and perhaps easier to add to, but that's not my department as such. =-)
Oh... can we have a preview button for forum posts too please?
i'm absolutely for transparent votes - and as Oswald said, if you don't stand by what you voted, it wasn't worth doing. |
| |
TDJ
Registered: Dec 2001
Posts: 1879 |
It's not that I don't agree with Oswald about the transparant voting - I just think it doesn't belong in this thread. First concentrate on the big picture, then on the details. And transparancy of votes is a detail, nothing more. |
| |
Nafcom
Registered: Apr 2002
Posts: 588 |
Quote: It's not that I don't agree with Oswald about the transparant voting - I just think it doesn't belong in this thread. First concentrate on the big picture, then on the details. And transparancy of votes is a detail, nothing more.
I totally agree with you again, stunning! =) |
| |
Oswald
Registered: Apr 2002
Posts: 5017 |
how about being ontopic, instead spamming the forum with these "dont be offtopic" "I agree" stuff ?
some more thoughts of the gui:
on the userpage:
- add information
- communicate
should not be there, instead they should be at the top tab amongst "userpage" "browse" etc... every now and then I have to think for long seconds until I realize that I can add a new entry or send a PM through the userpage. The userpage should only contain stuff related to the user.
browse page:
this one should be shown as an example on how to NOT to design a gui.
Takes a minute or two to realize what does what. An unexperianced pc user would be lost here forever.
The 4 browse category should be strictly displaced and separated, with their sub options. The options instead of icons should have a text describing what does that (tooltips help here, but you'll never learn which icon is for what). Even the term "browse" is confusing why not "search" ?
the forums:
this part is okay. only a few issues:
- why isnt the start new topic at the top of the page ? who wants to scroll all the way down to find it ?
- the small "my settings" link should be moved to the userpage as "forum settings" or something.
- what does the first column do in the topic list ? its all empty.
btw it would be nice to have a random logo like pouet... it would be cool if only c64 format would be accepted.
well this is just my 2 cents. As I have 0 experience in designing the inner roots of such a site, I just pointed out the first few things that came to my mind about the current design. |
| |
TDJ
Registered: Dec 2001
Posts: 1879 |
Oswald, how about actually trying to understand what it means to build a web application, instead of bothering us with those pesky details?
By the way, before you even started, I already said:
Quote:3. You shouldn't just change the way things look, the most important is the user interface. Make it easier for people to add stuff etc. like Hein proposed. Interaction Design is a must these days!
See? Requirements on a HIGH level. Software Development 101. |
| |
Oswald
Registered: Apr 2002
Posts: 5017 |
The Dark Judge: your personality is just as overgrown and childish as your handle.
Understanding a web application and pointing out bugs in an existing gui is not the same thing. or you think some1 not understanding the inner workings of something cannot recognize its bugs ?
"By the way, before you even started, I already said:"
you're so coool coz u posted sooner. are you serious?
"3. You shouldn't just change the way things look, the most important is the user interface. Make it easier for people to add stuff etc. like Hein proposed. Interaction Design is a must these days!"
am I not suggesting the same ? make easyer to use csdb.
Thats it about my user feedback here.
cya. |
| |
Hein
Registered: Apr 2004
Posts: 933 |
dont drag me into your arguments..
oh, I stepped in. why not sabotage other platform scenes to get over this 1 mhz trauma? |
| |
Matt
Registered: Apr 2002
Posts: 589 |
pah.
I'm really happy with the way the database like it is now.
but u r planning to make a v2 anyway, so don't make it too perfect bcos lot's of girls over here wouldn't know what to complain about anymore.
|
| |
Moloch
Registered: Jan 2002
Posts: 2893 |
Haven't read the entire thread... but I'm hoping that BBS entries will be included with v2... old thread, lots of people agreed it would be a nice addition.
Putting them in the trivia section now.
|
| |
Nafcom
Registered: Apr 2002
Posts: 588 |
Quote: Haven't read the entire thread... but I'm hoping that BBS entries will be included with v2... old thread, lots of people agreed it would be a nice addition.
Putting them in the trivia section now.
Agreed. |
| |
TDJ
Registered: Dec 2001
Posts: 1879 |
@Oswald: how clever, making fun of my handle. To my defense: I was 17 when I picked it, what's your excuse for being childish at age 29?
Also, please note that I never attacked you personally, I just tried to make you see the difference between just randomly shouting stuff, or actually doing this the right, organized way. And that goes for dealing with GUI issues too. Because believe it or not, web application is more than just 'the inner works'.
The reason why I quoted myself was not to look smart, but to show you that you just repeated something that was already said. No need for bringing it up again and again and again ..
@Hein: I never dragged you into this, I quoted something which just happened to feature your name. So do me a favor, and step out of it again, ok? ;) |
| |
Dipswitch
Registered: Apr 2004
Posts: 42 |
for hosting, you should check out http://www.untergrund.net .
it is a "sceners for sceners" website hosting service ran by scamp (breakpoint mainorga and vacsid coder). i have a website there myself, and the service is pretty reliable and supports all kind of css/php/mysql shizzle. |
| |
dalezy
Registered: Apr 2002
Posts: 475 |
now what would be if untergrund.net wouldn't support css. not to mention the scary fact if html wouldn't work there too. =) |
| |
Steppe
Registered: Jan 2002
Posts: 1510 |
Just imagine they're running a server without internet access. Now that's scary! |
| |
cba
Registered: Apr 2002
Posts: 932 |
So what's the deal about V2 ?
any news ? |
| |
Perff
Administrator
Posts: 1665 |
We are slowly starting up again now that the new server is in place. However it IS summer, so things might be moving a bit slow.
Anyone who wish to join the V2-crew can send me a pm where you write what you can contribute with (coding, design, ideas, etc..) |
| |
Sander
Registered: Jan 2002
Posts: 487 |
Hello, I'm Sander.
Joined the team yesterday, and will primairily focus on information design, interaction design, design / art-direction (and a general pain in the ass).
I want to emphasize i do not intend to actually design the new CSDB. (but willing to set out some styleguides and grids).
Hopefully later this week i'll read my way down to this post.
|
| |
MRT
Account closed
Registered: Sep 2005
Posts: 149 |
...primairily focus on...
He he heeee ha ha haaaa |
| |
TDJ
Registered: Dec 2001
Posts: 1879 |
Quote: ...primairily focus on...
He he heeee ha ha haaaa
Well, given that I am involved too, you can understand why the petname for this project is FSDb ;-) |
| |
Conrad
Registered: Nov 2006
Posts: 833 |
I know this is an old thread, but has there been any development for this idea at all? |
| |
instant
Registered: Mar 2020
Posts: 20 |
This thread seems pretty stale. I've got free cycles so I've started on things. If anyone is interested in helping please get in contact with me. Perff's help would be greatly appreciated of course (even at the bare minimum) but I understand if he is no longer interested in this.
Backend:
nodejs
appwrite.io
Middleware:
graphql
Frontend:
Nuxt3 or Flutter 2
(could do both actually. nuxt for web and flutter for mobile apps? or Flutter for both web and apps)
That is what I'm thinking. I'm open to other ideas but would like to stick to some of the latest tech. Then I'm learning something new that will benefit me with other projects while rebuilding everything at CSDb. |
| |
chatGPZ
Registered: Dec 2001
Posts: 11108 |
The most important thing is a proper redesign of the database structure, fixing all the little flaws we learned about in the past decades.
After a few iterations of design/review/fixing that - it makes sense to start thinking about redoing the website :) |
| |
instant
Registered: Mar 2020
Posts: 20 |
I'm thinking a NOSQL database would be best. The current database seems over engineered. JSON documents would work just fine and then maybe a small database just for the indexes for searching. Appwrite.io is perfect for this. |
| |
Digger
Registered: Mar 2005
Posts: 421 |
Another thread from the grave. I think I've posted it already years ago – the only way IMO to move forward is to make CSDb open-source and let ppl fix issues, there's a few of us knowing modern JS stacks.
Ofc there needs to be a team to approve PRs. |
| |
chatGPZ
Registered: Dec 2001
Posts: 11108 |
As someone who has seen the source i can assure you: there is no way to fix all the mess that needs fixing in the existing codebase. In any case, the biggest and most important task is to create a sane db structure that is future proof and migrate the data to it. Making the frontend to that db then is just the icing on the cake. |
| |
Seven
Registered: Jan 2002
Posts: 201 |
Tech stack first, concept second. I see we're in good hands. |
| |
instant
Registered: Mar 2020
Posts: 20 |
I'll have something worked out to show soon. PM me if you are interested in helping. :) |
| |
JackAsser
Registered: Jun 2002
Posts: 1989 |
Quote: The most important thing is a proper redesign of the database structure, fixing all the little flaws we learned about in the past decades.
After a few iterations of design/review/fixing that - it makes sense to start thinking about redoing the website :)
@instant: Really listen to this.
If I were you I'd start interviewing the main contributors to the Database, like Hedning, to properly understand what's needed for preservation and history.
Once you have a proper design, do an API to expose it. You can for sure use GraphQL or slap GraphQL onto a REST API, whatever floats your boat.
After that, think about UI tech choices. |
| |
Burglar
Registered: Dec 2004
Posts: 1031 |
If your only tool is a hammer, every problem looks like a nail. |
| |
chatGPZ
Registered: Dec 2001
Posts: 11108 |
Another important thing to consider is: There must be multiple persons who know all the tech and can expand and maintain it. If you are the only one who knows a certain piece of the stack, it is doomed to fail on the long run. |
| |
instant
Registered: Mar 2020
Posts: 20 |
I appreciate the feedback. I'm working with the information available to me. If Hedning or others want to chime in that would be great.
Believe it or not I've been doing successful software and hardware development and managing development teams for the past 40yrs. I'm confident in my ability to make stuff happen. ;)
This is fun for me. If others want to help, I'm happy for them to join in. |
| |
Youth
Registered: Aug 2003
Posts: 40 |
I have quite some experience in building enterprise software and would, like Groepaz, advice to do a bottom up approach, focusing on the database. Gather information about how the users use CSDB, and how they want to use it. See how the current model fits those use cases or make a redesign.
Then think of a way to disclose that information with an API that is specific enough to support the envisioned use cases, but flexible enough to discover new ones.
Then build a UI on top of that API with a well thought out interaction design. Then paint it a nice color and smooth out the cracks.
This sounds like building a house, but unlike building a house, I would do it in small iterations with a short feedback loop, by having something simple that works and can be tested early on.
At the bottom use proven database technology. An SQL database like Postgres might sound old fashioned, but it's very mature, robust and has an immense user community.
For the rest of the stack, use technology that has some track record, is well maintained and has a big user community. REST/JSON or GraphQL are obvious choices for the API. Decide if you want a single page application or a (regular) multi page application and choose an established frontend technology based on that.
Keep in mind who will be working on this. To keep this project sustainable over time, you need a team of developers, with people swapping in and out. So open source it, with a small team maintaining control over the core design and the process of accepting changes from other developers.
So TLDR; Focus on data and user interaction with the data. Choose a tech stack that is modern enough to not get outdated soon, but mature enough to be stable, well maintained and has a large community to support it. Build a developer-friendly work environment to build and maintain the software over the long run.
N.B. this is a first draft of how I would approach it. Pick, mix or ignore at your own will ;) |
| |
Mr. SID
Registered: Jan 2003
Posts: 421 |
Obviously the only way forward is to merge with GameBase64 and switch to a Microsoft Access database… |
| |
Elder0010
Registered: Apr 2011
Posts: 7 |
Quote: I have quite some experience in building enterprise software and would, like Groepaz, advice to do a bottom up approach, focusing on the database. Gather information about how the users use CSDB, and how they want to use it. See how the current model fits those use cases or make a redesign.
Then think of a way to disclose that information with an API that is specific enough to support the envisioned use cases, but flexible enough to discover new ones.
Then build a UI on top of that API with a well thought out interaction design. Then paint it a nice color and smooth out the cracks.
This sounds like building a house, but unlike building a house, I would do it in small iterations with a short feedback loop, by having something simple that works and can be tested early on.
At the bottom use proven database technology. An SQL database like Postgres might sound old fashioned, but it's very mature, robust and has an immense user community.
For the rest of the stack, use technology that has some track record, is well maintained and has a big user community. REST/JSON or GraphQL are obvious choices for the API. Decide if you want a single page application or a (regular) multi page application and choose an established frontend technology based on that.
Keep in mind who will be working on this. To keep this project sustainable over time, you need a team of developers, with people swapping in and out. So open source it, with a small team maintaining control over the core design and the process of accepting changes from other developers.
So TLDR; Focus on data and user interaction with the data. Choose a tech stack that is modern enough to not get outdated soon, but mature enough to be stable, well maintained and has a large community to support it. Build a developer-friendly work environment to build and maintain the software over the long run.
N.B. this is a first draft of how I would approach it. Pick, mix or ignore at your own will ;)
I could not agree more.
At the cost of sounding rude, I also suggest to use stable and mantained technologies. Ignore fanciness and go for mature solutions, if you don't want to trash the whole project after 2 years or hire a full time developer to mantain it. Basically any "super cool js thing that does something already existing since 30 years ago in a different way because js is cool and you're old" must be avoided.
Mysql or Postgres can handle *millions* of records, and since the nature of the data here, the only way to go is to use a relational approach (if you don't want to lose your data).
As a matter of personal taste, I suggest also to take a look at django framework. With djangoREST you can easily build up an api. |
| |
chatGPZ
Registered: Dec 2001
Posts: 11108 |
youth+++ |
| |
instant
Registered: Mar 2020
Posts: 20 |
The goal is to not recreate the same thing with the same problems.
It should be as simple and flexible as possible.
It should be easy to maintain.
I don't think the tech that is selected should cater to the least common denominator. Anyone wanting to help should have a desire to learn some new things and be savvy enough to figure it out with the help of some good documentation and well commented code.
I need to do more testing but I think I've got most of the backend tech and database worked out already. It will be flexible enough to extend data easily and handle use case exceptions that may crop up.
A vague idea of the user interface needs to be kept in mind while working on things but that is not important until the data and backend/middleware are functional. My priorities are:
1. Make it work! It has to be accurate first or it is worthless.
2. Optimize! Make it as efficient as possible. This is an ongoing iterative process that will be revisited over and over during the life of the application.
3. Make it pretty! This is where all of the fancy UI stuff is worked out. Also an iterative process. Most important information first. Organize the display of the information and making it pretty. Extend UI functionality as we go where it makes sense.
4. Enhance with extra services. API for external applications / RSS feeds / Webhooks / Integrations
People keep saying I need more in depth scene and user interaction knowledge. I totally agree. I'm working based on the information I have at the moment. Mostly from my own experience with the site, what I have read here in the forums from others, and experience from similar projects over the years.
Who wants to dig in and interview Perff, Hedning, and whoever else to define some requirements and use cases? I'm happy to do that if they will get in contact with me. I have recently reached out to Perff but have not gotten a response. I'll contact Hedning.
Everyone is happy to tell me what I need to do or tell me what they think I'm doing wrong but no one is stepping up to offer to help. This thread has been dead for years. I don't mind taking the lead on this. I'm going to do it with or without anyone's help. I'd prefer that others join me though. It will be more fun.
Contact me and let me know what you are good at and what you want to help with.
PM here or idolpx!ircnet / idolpx!efnet |
| |
chatGPZ
Registered: Dec 2001
Posts: 11108 |
Sounds like another one-man show with no future beyond the interest of the one single dev. |
| |
Youth
Registered: Aug 2003
Posts: 40 |
Quoting instant
Everyone is happy to tell me what I need to do or tell me what they think I'm doing wrong but no one is stepping up to offer to help.
Sorry about that, I am definitely interested in helping out. PM sent. |
| |
instant
Registered: Mar 2020
Posts: 20 |
Quote: Sounds like another one-man show with no future beyond the interest of the one single dev.
It's only a one-man show if you or others aren't interested in helping.
You know these guys better than me. Can you get the info we might need?
I'll get a repo setup and share what I've been working on soon. |
| |
chatGPZ
Registered: Dec 2001
Posts: 11108 |
I have tried to help, but it was all dismissed. It seems you are already three steps further than you should be at this point.
As soon as a database schema is posted for review i (and i am sure some others) will look at it, no problem. I wouldnt write a single line of code before this happened and went through a few iterations.
And then we can try to find a team of people who work out what tech to use. As others already said, i'd go for a very conservative yet stable and well known sql database. Because this is what people know and can handle easily. |
| |
instant
Registered: Mar 2020
Posts: 20 |
This thread is 16yrs old. A thousand things could have been tried by now. I think I'm moving at just the right speed.
I'm still working out some of the data aspects. It will be simple and easily extendable. I think people will like it.
The UI/UX stuff could be worked on independent of the data though. Who wants to help with that? We just need to define all use cases we can think of.
How do most people use the site?
When someone identifies incorrect data, what should be the process for making the correction?
When adding a new release, is there a proper order for entering the data?
etc... |
| |
Raistlin
Registered: Mar 2007
Posts: 553 |
I would say that you should just do it .. bearing in mind that if it was every to replace CSDb, it needs to of course take all CSDb's data - assuming that Perff is happy with that? And then add to that with all the tagging stuff that Groepaz has been asking for .. and more.
Graphically, I'd be careful .. try to make it look too good and you run the risk of it becoming less functional. The design right now is old school, yes - but it works. I know a lot of people who like it exactly how it is. Particularly at most of our ages, change isn't always welcomed ;-) |
| |
Burglar
Registered: Dec 2004
Posts: 1031 |
@instant: while you talked on irc to groepaz, count zero and myself about your plans, you immediately dismissed all our tips without argumentation (except some arguments of authority that have zero substance).
We see the exact same thing happening in this thread: you are dismissing every valid response from persons with tons of knowledge of csdb, software engineering, architecture and project management. And again, without argumentation.
This is off-putting and most likely the reason why people are not jumping aboard. At least for me it is. |
| |
Youth
Registered: Aug 2003
Posts: 40 |
Reading this thread it seems that this is not so much a technical problem, as developing a sense of shared ownership and vision. I think a tech-first approach very much risks people not being motivated to join because they don't feel like they are part of the fundamental decisions made about the future of CSDb.
How about collecting the issues and wishes for the current site, have people vote, comment and have a small team classify and prioritize what needs to be done. If you open a Github project or something similar, it's pretty easy. |
| |
chatGPZ
Registered: Dec 2001
Posts: 11108 |
Indeed. The tech itself is irrelevant mostly - except that it should reflect what people know and can help with.
The important part is the database structure, and careful planning what goes where - so you don't end up with the same patching on patching on patching madness that csdb currently is. As said before, step one is designing the scheme and putting it up for review so people can tell what is bad about it and what needs to be added.
And even more important than the most important thing is that its driven by at least a few people from the currently active csdb community (i hate to use that word), ie the people who submit and maintain the data behind the scenes. Without those people, you'll end up with a pretty modern site which noone wants to use. |
| |
JackAsser
Registered: Jun 2002
Posts: 1989 |
Out of curiousity it would be nice to see the current DB scheme for CSDb v1. |
| |
instant
Registered: Mar 2020
Posts: 20 |
Quote: @instant: while you talked on irc to groepaz, count zero and myself about your plans, you immediately dismissed all our tips without argumentation (except some arguments of authority that have zero substance).
We see the exact same thing happening in this thread: you are dismissing every valid response from persons with tons of knowledge of csdb, software engineering, architecture and project management. And again, without argumentation.
This is off-putting and most likely the reason why people are not jumping aboard. At least for me it is.
Most of the "tips" were negative judgements and ridicule regarding my ideas for how I envisioned it working. Also it seemed that there is an assumption that I have not put any thought into the data schema and other steps required to take on a project like this. All of that is fine. I don't recall any arguments from authority. Someone did ask about my experience working on big data projects and I rattled off some of the projects I have been involved in.
I didn't mean for it to come off as dismissive. I appreciate all feedback. But when someone says "Oh that won't work. You need to do this." and doesn't explain why it won't work or doesn't offer to do any of the work they are suggesting, that is dismissive and not helpful. It takes no effort to shit on something. That is when I say "Thanks" and continue on with my plan. I'm not here to argue. This should be fun.
I have put a lot of thought into this before even mentioning it on irc or here. I agree that the data is very important. I think that people are making it more complex than it needs to be.
I think I have some ideas that will work. I'm putting together a proof of concept so that maybe it will better convey what I'm trying to do rather than people just assuming that I'm already fucking up.
I'm trying to avoid analysis paralysis. I'd rather do something, anything, to get things moving.
@Groepaz: Could you be more specific? Who are the people we need to get involved? |
| |
Shine
Registered: Jul 2012
Posts: 327 |
@instant:
Your project is brave in my eyes. (And long awaited by (at least) some people!)
You have to understand, that $ALL of this kind of ideas resulted in hot air in the past (sadly).
And if poeple don't give feedback or even a reply, you can be sure, that they are simply not interested in your project.
Anyways, keep on working and surprise us with your innovative ideas and executions. |
| |
chatGPZ
Registered: Dec 2001
Posts: 11108 |
Quote:could you be more specific? Who are the people we need to get involved?
You already talked to half of them and were not listening. The other half can speak up if they want to - but knowing them, that will not happen, because they prefer to work behind the scenes. Those people will follow when the other half has actually made something worth talking about.
Quote:Most of the "tips" were negative judgements and ridicule regarding my ideas for how I envisioned it working.
When everyone tells you a relational db is what it needs, because the data is highly relational - and you dismiss it and in the next sentence say you need to do more research... yeah that is what will happen. It makes it look like you are not planning to listen to the people who actually know all the things you still need to research. |
| |
instant
Registered: Mar 2020
Posts: 20 |
@Shine: Thanks! I hope to learn some stuff in the process and want the end result to be something that people will love as much as all of the great work that has been done to date. :) |
| |
Burglar
Registered: Dec 2004
Posts: 1031 |
Quote:But when someone says "Oh that won't work. You need to do this." and doesn't explain why it won't work on the contrary, I gave you many explanations as to why NoSQL is a bad choice for this type of highly relational data.
I also offered my help, just not on a appwrite/nodejs/nosql solution. It is not the right hammer for this nail.
We also explained that your approach would require all relations to be coded into the app-layer, that on change, cascading updates need to be scattered around the nosql store. And I didn't even mention search yet :)
This is why everyone in this thread is suggesting a different approach: start with gathering requirements and the database design, bottom-up. Get people like perff, hedning, count zero, etc involved. Without them, any project aimed to replace current csdb is bound to fail.
That said, your enthusiasm is great :) Whenever you need feedback on your plans, you know where to find me, I'll give it to you bluntly :) |
| |
instant
Registered: Mar 2020
Posts: 20 |
I will have more posted soon but let's get things moving.
Please add an issue to the repo for each individual requirement, feature, suggestion, etc.
https://github.com/idolpx/csdb-ng |
| |
spider-j
Registered: Oct 2004
Posts: 444 |
Having patched a lot of "unpatchable" things in my job I wonder why if it is not possible to open source CSDb "as it is" and then have people trying to make it better. I understand that sometimes it is good to "start from scratch", but I think even with the most ugly codebase there will be room for improvement if more people would look at the code and reflect about it. Especially when it comes to "visual" stuff which happens to be one of the first "issues" in instants bug tracker: "make it mobile friendly". This is really "simple" front-end stuff which could be achieved without touching any of the discussions which structure the database should have.
I doubt that "one lone coder" effort will result in "CSDb v2" or something like that. Gradually improving what is already there may sometimes be a pain in the ass, but I can't believe that it is a really impossible task. |
| |
Youth
Registered: Aug 2003
Posts: 40 |
I agree with spider-j
I have been looking at the current site trying to find issues I would like fixed, and a few things struck me:
- It basically works for me as a user. I browse stuff, find stuff, download stuff and occasionally interact on forums, place comments and upload my own stuff. Apart from the design and interaction feeling a bit old-fashioned, nothing frustrates me.
- It is big. It has evolved to have lots of functionality that is actually being used.
- It has been around and in use for a long time. Things are being added and commented upon daily.
- It seems to be always up and performance is ok.
- My main point of improvement is to have a mobile UI. Read-only is fine for now.
Rebuilding a system like this from scratch is almost never a good idea. It might need some serious refactoring to enable future improvements and keep/make the site developer and maintainer friendly though. |
| |
Frantic
Registered: Mar 2003
Posts: 1627 |
#donttouchmycsdb |
| |
Compyx
Registered: Jan 2005
Posts: 631 |
How likely is it that the source code of current CSDb will ever be made public? Same with the underlying database. I have a feeling the answer would be "you can pry it from my dead, cold hands". But I'd like to wrong on that one ;)
As for starting from scratch: sometimes that's the way to go, (try to) implement things properly from the start. From what I've heard the current code is hack upon hack upon hack. |
| |
Oswald
Registered: Apr 2002
Posts: 5017 |
Quote: I will have more posted soon but let's get things moving.
Please add an issue to the repo for each individual requirement, feature, suggestion, etc.
https://github.com/idolpx/csdb-ng
I have never worked on such a project but some things about the db scheme seems not right to me.
dates are strings and ints, sometimes a date is just a year or a month, shouldnt all be date type ? (if db engine supports it)
scener has both handle string and pointer to handle entry, why not just the pointer? |
| |
iAN CooG
Registered: May 2002
Posts: 3132 |
Quote: I have never worked on such a project but some things about the db scheme seems not right to me.
dates are strings and ints, sometimes a date is just a year or a month, shouldnt all be date type ? (if db engine supports it)
scener has both handle string and pointer to handle entry, why not just the pointer?
Dates are often partial, some times only year is known, there are even cases where day & month are mentioned but year is not. Better leave the dates fields separate. |
| |
Oswald
Registered: Apr 2002
Posts: 5017 |
google tells me there are better ways to represent uncertain dates
https://stackoverflow.com/questions/27726716/how-to-represent-d..
date should be just date, and then another field describing the fuzziness. |
| |
spider-j
Registered: Oct 2004
Posts: 444 |
I agree with Oswald. Date in a database should be date / unix timestamp. If you don't need all details of it in the frontend: don't use it. But if needed later on you will always have the flexibilty to extend it without changing the database structure. |
| |
Raf
Registered: Nov 2003
Posts: 343 |
Quote: How likely is it that the source code of current CSDb will ever be made public? Same with the underlying database. I have a feeling the answer would be "you can pry it from my dead, cold hands". But I'd like to wrong on that one ;)
As for starting from scratch: sometimes that's the way to go, (try to) implement things properly from the start. From what I've heard the current code is hack upon hack upon hack.
True, from my playground: back in time (10+ years) there was custom coded forum on c64power.com which I've tried to make better but finally I ended up exporting entire database using my own written tools to MINIBB forum engine and when it was like they charged money for many things that were available right from the start on other solutions entire forum was auto-converted using someone's else script to SMF (and it's still functioning at http://c64power.com/forumng/ [ng was meant for new generation ;) ])
Actually it's better to define new (proper and extensible) database right from the start with decent MVC code and convert existing data than dig in knee deep hacks which arose during 20 years. |
| |
TheRyk
Registered: Mar 2009
Posts: 2062 |
Quoting spider-j"make it mobile friendly". This is really "simple" front-end stuff which could be achieved without touching any of the discussions which structure the database should have.
...
I've seen database stuff turned mobile-friendly ending in some front-end that might work on mobiles but is hell to use on real hardware (here meaning browsers running on computers, not smartphones). And once enough hipsters fall in love with that shiny new stuff, the old stuff ain't working anymore on old frontend and no-one cares to update it anyway as it's old...
No, thanks, but I'm with Frantic here :)
#donttouchmycsdb
The few things I'd like to change wouldn't need re-inventing wheels or starting from scratch but rather turn the wheel back when it comes to certain whacko (sub)categories or flags
But this is only me - grumpy old Ryk, not an official Mod statement |
| |
Monte Carlos
Registered: Jun 2004
Posts: 351 |
I see that the csdb is in it's 20th year and as we know from old discs there will be times when everything falls apart. This is not the case currently, however I can understand the maintainers doubts about keeping csdb V1 until it is too late. I think it is very foreseeing to work on csdb V2 as long as csdb V1 is still in place. Nobody wants to shut down csdb V1, instantly. |
| |
instant
Registered: Mar 2020
Posts: 20 |
Quote: I see that the csdb is in it's 20th year and as we know from old discs there will be times when everything falls apart. This is not the case currently, however I can understand the maintainers doubts about keeping csdb V1 until it is too late. I think it is very foreseeing to work on csdb V2 as long as csdb V1 is still in place. Nobody wants to shut down csdb V1, instantly.
I figured it would run concurrently to get everything up to par. It would sync every hour or so to stay up to date with newest entries.
Later when everything is solid and tested we can turn on the ability to submit new entries. Even then CSDbv1 would still operate as usual but it would not get any updates that were made only at CSDbv2.
If it takes a few months to work out the details and get things going that still might be considered an "instant" in contrast to the time of the first post in this thread. :) |
| |
bOOZElEE
Registered: Dec 2002
Posts: 34 |
All credit to your enthusiasm, but I think you're underestimating the complexity a bit. very. ;)
But don't get me wrong or even let my words lead to slowing down your eagerness.
Some words about tech-stack (despite the usual "concept first, tech-decision second"):
In my eyes considering the approach of implementing CSDb V2 as appwrite/nodejs/nosql clutter is very questionable: not only does a new (rotten) JavaScript framework (react, vue, whatever) come out every few years, which is then hyped as the new hot shit (pardon my french), but also node itself is a filthy juggernaut: you have such a dependency hell, countless packages are pulled/installed, some (if not most) of those no one ever looks at or let alone review their source-code. (Just pops up in my mind: in 01/2022 the node libs 'color' and 'faker' were on purpose rendered useless by the author resulting in thousands of production systems being broken 'relying' on these libs). No one wants to maintain this.
CSDb V2 is not intended as another short-lived customer fancy-app project.
If CSDb V2 should be up and running for the next 20 years, drop the idea of using nodejs-crap. Well, everything that comes after 20 years won't interest most of us anyways, but nevertheless someone wants to/should maintain the stuff up to then or maybe for a few years more ...
Regarding database: due to the necessary relations, nosql imho would be bad choice. Better use a 'real' database like postgresql, mariadb or something like that. But some other people pointed this out already.
As there are a lot (and by lot I mean a lot) of assets aka releases of all different types available (download, tagging, etc.), also a proper File Abstraction Layer (FAL) is necessary. This needs some brainz. Huge brainz.
Even as the db-scheme is not yet ready, nevertheless I'll just throw in the following as a possible tech-stack
(In hope of earning another +1 for 'Tech stack first, concept second.' chrchr):
Haskell in combination with Yesod
"Haskell is a powerful, fast, type-safe, functional programming language."
Granted the learning curve is steep and hence many people don't work with it, but it's miles apart compared to php-fumbling or even js/nodejs-crap and once you're "in it" you will appreciate the benefits.
Just my personal opinion though, but Haskell should be worth evaluating when time has come to decide on tech-stack.
Just my two Złoty |
| |
Skate
Registered: Jul 2003
Posts: 490 |
I once offered help for CSDb V2 and got declined. I'm fine with it. But reading some of the discussions, i'm glad i got declined. :) Pleasing a large community is always a difficult task.
About V2 db being sql/nosql, why not both (if required)? I use sql/nosql side by side. Almost every system have some audit logs, sessions etc. Nosql is very suitable for these tasks. About rest of the complex query requirements, relational databases save a lot of time and have good performence.
I agree with bOOZElEE on not necessarily using the state of the art technologies which would be a shame to use in a few years according to newbies. :) But i disagree with "nodejs-crap" kinda words. They all have their place. For instance i use PHP for more than 20 years now. But when a realtime connection (Web Sockets) required, i don't try to solve it using PHP. Node.js is very suitable for the task.
Of course you can use anything, PHP, node.js etc are not the real subject here. But most of the time people choose either one of them. I use their powerful sides together. It applies to backend, frontend, database, everything... |
| |
instant
Registered: Mar 2020
Posts: 20 |
Ok... I've staged an Apollo GraphQL server with my first pass at building the required queries and resolvers. This is read only at the moment. It may be missing some data and other items might need to be renamed or rearranged to make more sense. The goal is to be able to pull exactly what is needed for any page.
https://csdb.idolpx.com
Please play with it and leave me some feedback here or hit me up in IRC. I believe I built things in a way that we can get at any data across all relationships. To avoid duplicating data, anytime a single ID or array of IDs for a given entity (Releases, Groups, Sceners, Handles, Events, BBSs, SIDs) is referenced it loads a single unique copy of that data and adds it to a base array in the query response. So in instances where a single handle or group is referenced multiple times for multiple things, only the ID is listed in the nested data. The detail can then be looked up by ID in the base array.
All of this is done using only JSON documents. No database yet.
Next I will start working on the mutations. That is going to be interesting. |
| |
Count Zero
Registered: Jan 2003
Posts: 1821 |
Quote:
https://csdb.idolpx.com
borked after a week already or what is this?
But since the original topic and intention here is hardly being met anhow - closing to avoid further resurrection. |
