Author Topic: 1000 categories and resource memory and time out issues  (Read 4573 times)

0 Members and 1 Guest are viewing this topic.

Offline surferboy

  • Full Member
  • ***
  • Posts: 142
    • View Profile
1000 categories and resource memory and time out issues
« on: July 17, 2012, 09:18:31 PM »
Hi -

We are a devoted 4images user. For our forum, we use Invision Power Board. Recent updates on their upcoming launch of their image gallery indicates that sub-albums will not be permitted while sub-categories will.  Given the structure of 4images, would the issue pasted below happen to our 4images installation, where we eventually exceed 1000 categories, be it regular or member categories?   Please note that resource issue that IPS describes is on a user's hosting platform and not their own hosting platforms.

 The IPS announcement and explanations are listed here:

In IP.Gallery 5.0, you will no longer be able to create sub-albums.  Sub-categories are still supported, however albums themselves can only be placed within a category, not within another album. ....These changes represent necessary deviations from past versions to allow IP.Gallery to scale effectively as the site becomes larger.  One of the primary issues faced in past versions was resource consumption as the number of albums on the site grows - this will no longer be an issue with specific rules in place that prevent complex hierarchy challenges.

Firstly, we are not removing subforums (however, we have worked on MANY sites where the admin has added over 1000 forums and suddenly they get memory and timeout issues because it takes so many resources to build the forum tree when they do this).  The difference with Gallery is that you WILL eventually top 1000 albums if your Gallery continues to grow.  This is because users create the containers, not the administrator, so mechanisms have to be built in to account for this.

Once you start reaching around 1000 albums or more when they can be structured in complex hierarchys, there is no resource efficient manner of building the tree.  For instance

Category A
--Album 1
----Subalbum 1
------Sub-subbalbum 1
------Sub-subbalbum 2
----Subalbum 2
------Sub-subbalbum 1
--------Sub-sub-subalbum 1
----------Sub-sub-sub-subalbum 1

In this scenario, to show the latest images (that you have permission to see) for any parent container, you need to loop through your children containers (the ones you can see) to figure out what their latest images are.  The only real way to do this is to load all of the albums from the database, put them into a tree structure in php, then loop through them.  Pulling all of the albums when you only have 100 isn't an issue.  Pulling all of them from the database when you have 10,000 is.  And no, there's no realistic way to pull only the albums you need because you don't know which ones you need (you have to recursively iterate through the albums to figure out what each parent's is and to determine if it is one you need to check).

Thanks to all the php programmers out there who understand php structure and can explain how or why 4images does or does not face the same issue.